I really wish managed airflow instances were cheaper for smaller companies. I built my own using spot instances and it's so affordable compared to astronomer and the others.
So far it's been very low maintenance - outside of the few random scares where dag logs filled up my server - but then I researched and found maintenance Dags that prevent a lot of issues.
Wondering when my next outage will be is always fun, but it's been pretty stable so far.
E: I know and appreciate the tech they put into it. It's just too high of a price for me once I get the workers added. I still want migrate mine to fargate workers at some point though.
I just glanced at our own airflow instance in AWS (not on this service). We run 1 t3.xlarge instances 4vCPU for the scheduler and web server and 1 t3.xlarge instance (4vCPU) for the workers. At $0.33 per hour (on demand), this seems to most closely match the resources for their medium or large offering, at $0.74-$0.99 per hour (roughly 3x).
I realize you are buying not just the compute, but the management, but that ends up being something in the cost range of $300-$500 or so per month for the airflow management part of it. Seems a bit steep. $50-$100/mo would be a no brainer for us. For some orgs I can see this being a great solution, but its not really friendly for the little guy (with a min price of $350/mo).
Would you agree the $300-500 is easily offset by any 1 production incident/outage that would require manual intervention on the airflow servers (and thus developer salary for however many hours to fix, and lost productivity elsewhere)?
I understand that the premium is paid _every month_ -- and you may not otherwise have an incident every month -- but the AWS premium can also be considered an _insurance premium_ against those outages.
I used to manage an airflow deployment (of which my team was the primary consumer), and it was not enjoyable in the least.
not OP, nor do we use airflow (not sure there is a fit for us) - but $300-500 for something that could cost our modest company an equivalent half day of human time to mitigate, with possible tens of thousands in lost revenue if it occurs at the worst possible time, seem like a win-win peace of mind proposition.
(I'm not the OP)
For an enterprise, this pricing would work for some projects, for the reasons you suggest. Although this assumes nothing ever goes wrong with the managed offering, which is unlikely.
But for everyone else, it's a hard sell, especially in these days of infrastructure-as-code - even if you had to rebuild an Airflow server from scratch, it's not going to take very long.
(am OP) We haven't had a major outage in 3 years of using airflow. Some issues, and it consumes some time, but so would managed airflow in all likelihood. Most issues are related to "airflow configuration type things" that presumably this solution would not fix.
I am really just surprised it is not a "no brainier price". It's a tough sell. I am sure it's valuable for some people, I just can't really justify using it. I think value to me is like $100/mo (as random airflow user on the internet).
But why Airflow, it has so many weird things. I hope it is dethroned soon.
Because people use it. It's been around a while and has an ecosystem around it. (This is the interesting thing about AWS, they keep the focus on what people do use, rather than some opinionated idea of what they should use.)
fyi airflow 2.0 ships in the next few weeks, we're made a lot of improvements https://www.youtube.com/playlist?list=PLCi-q9vYo4x-PESoBcXN0...
I am hoping Argo Workflows (https://argoproj.github.io/projects/argo) will make that happen. It decouples orchestration from data flow nicely and runs on Kubernetes so is highly-available.
I like Argo and have used both solutions, but I'm not yet convinced that YAML based workflows are superior to Airflow's Python Code as workflows.
Agreed. We've had good luck running all sorts of jobs on Argo and wrote up some of our experiences at https://www.interline.io/blog/scaling-openstreetmap-data-wor...
Agreed. It's the golden standard for data workflows right now, but it's pretty cumbersome to work with.
I'm building a SaaS product to try and fix the issues I found with Airflow et al. Feel free to check my profile/reach out if you're interested in trying an alternative.
I wonder how it compares to astronomer.io, and Google's managed airflow thing.
i'd let you know if their service came up :) been waiting an hour after hitting the "create" button
disclosure: co-founder of astronomer
service never came up; also can't delete it :(
"Environments with CREATING status must complete previous operation before initiating a new operation."
can't email support for help (i only have basic plan)
any AWSMWAA ppl on this thread and can help? the instance name is `airflow-ry-test`.
I think you'll need the account number for them to find out :P.
Thank you AWS. I think they just saved me $85K.
firing your devops person? maybe wait a few weeks to be sure!
There are now at least 4 different implementations of every data/app related technology: the oss/original version, the aws version, the azure version and the gcp version
Is this a good idea? I donāt think so
I respectfully disagree. Not only do cloud native versions of an OSS encourages innovation and enrichment, it also lowers the operational cost of running it in-house and aligns closely to Devops principles.
I haven't tried it yet but this honestly looks pretty close to vanilla, the post even says that they contributed their patches to upstream. What differences have you spotted?
Of course their storage backend is going to be S3 and they are going to send logs to Cloudwatch, we have been doing it in a similar way for quite some time and it is what I expect from a solution managed by AWS.
> Is this a good idea? I donāt think so
I agree.
It would be swell if, as part of the bigger open source movement, Jeff, Bill, and Larry - who at their disposal have AWS, Microsoft, and Google - competed with one another to submit the meritocratically superior implementation of our meritocratically determined open source standards, and then we, as custodians of the open source project, would select the winner, merge it, and use it - to the exclusion of all other implementations - not because of bias, but because of the truth underlying our meritocracy.
The question for me is - when we engage in said meritocracy, with all these BDFLs calling the shots, will they be the ones to negotiate with Jeff, Bill, and Larry? Or will we?
Only so long as you are able to easily interop one tool from one cloud with the others without major consequences. If that was true, then you can freely mix and match cloud services from different vendors.
Alas, this relies on many things, not the least of which are peering agreements between cloud vendors that do not punish the consumer for using a service on one cloud with services from another.
It is possible to an extent today, but for competitive reasons, cloud companies do not seem to have a built in incentive to collaborate. Perhaps the cost of switching and lock-in will force cloud providers to work together in the long one, else they risk alienating customers, and then lack the ability to capture a larger market.
In other words, competition is good as long as people can realistically take advantage of it. I think cloud vendors have an obligation to see that this competition is encouraged and supported.
Get a daily email with the the top stories from Hacker News. No spam, unsubscribe at any time.