TLDR

Harnessing the power of AI can be magical and impactful. Before mastering this ability, one must understand the basic foundation of AI. We’ll go over basic math concepts, common prediction types, and popular algorithms.

Math

• Vectors

• Matrix

• Cost function

Classification and regression

• Accuracy

• Precision

• Recall

• Mean absolute error

• Root mean squared error

Algorithms

• Linear regression

• Logical regression

• K-Nearest Neighbors

• Decision trees

• Random forest

• Support vector machines

• Neural network

  Math

  

  Mathematical equations lay the foundation for AI. For basic applications of AI, being a math expert isn’t necessary as there are plenty of tutorials and pre-built libraries maintained by the open source community. However, learning some math concepts will help you understand why an algorithm works the way it does and how it can be improved.

  Vectors

  Vectors are an array of numbers (x = [1, 3, 5]), compared to a scalar which is a single number (x = 5).

  A vector is a mathematical way of getting from one point to another. Vectors need to travel some distance which is measured in magnitude (length) and direction (orientation).

  Vectors are used in machine learning for their ability to organize data. One example of a commonly used vector in machine learning is a feature vector. These contain information on multiple elements of an entity; for example, a feature vector can contain 5 values: the age of a user, the gender of a user, their height, their weight, and their current city. Feature vectors represent attributes of an entity in a way that a machine learning model can easily perform calculations on.

  Vectors are written into Python using Numpy. They can be created using the function np.array():

  # Load Numpy module
  import numpy as np
  
  # Creating a 1-D list (horizontal)
  list1 = [2, 3, 5]
  
  # Creating a 1-D list (vertical)
  list2 = [
      [20],
      [30],
      [50],
  ]
  
  # Vector as row
  vector1 = np.array(list1)
  
  # Vector as column
  vector2 = np.array(list2)

  Vectors can be added, subtracted, divided, multiplied, etc.

  Matrix

  

  A Matrix is a 2-dimensional array (table) of numbers with at least one column and at least one row. The array in a matrix can be represented as having m columns and nrows; therefore, our matrix is size m x n. A vector can also be considered a matrix. Row vectors can be represented as 1 x n, and column vectors as m x 1 .

  A matrix is measured by its size which is the number of rows by the number of columns.

  Matrices are used to represent data that machine learning models learn from. By structuring the data in a matrix, machine learning algorithms can leverage linear algebra and matrix multiplication for efficient calculations across large amounts of data.

  In Python, a matrix is represented by a 2-D Numpy array:

  import numpy
  
  input_arr = numpy.array([
      [10, 20, 30],
      [40, 50, 60],
  ])

  Like vectors, matrices can be added, subtracted, divided, multiplied, etc. Matrices of the same size are added together by adding the corresponding data points.

  Cost function

  Cost function is used to optimize a machine learning model to make better predictions. The cost function calculates the error between predicted outcomes compared with actual outcomes. The goal of training a machine learning model is to minimize the error from this cost.

  To find the difference of one datapoint: Cost = ( y − ŷ )2

  y = actual value
  ŷ = predicted value

  To find the difference of all data points: 1/2m x Σ( y −ŷ )2

  m = number of data points
  y = actual value
  ŷ = predicted value

  Gradient descent

  To minimize the cost function error so a line best fits the dataset, we need to use a gradient descent algorithm. Gradient descent is an optimization algorithm that finds the minimum of a function — finding the value that’ll give us the lowest error.

  

  Gradient descent works by taking steps to find the minimum value of the parabola graph.

  As the graph slopes down towards the x-axis, theta (measure of an angle) will increase as the negative slope is turned positive by alpha. As the graph goes up past the minimum, theta will become negative, signaling it has passed the minimum and needs to backtrack. Repeatedly running this algorithm will leave us with the minimum.

  

  Classification and Regression

  There are 2 main categories that most machine learning problems belong to: classification and regression.

  Classification is the process of taking an input and mapping it to a discrete label. Classification problems typically deal with classes or categories. For example, a photo of a cat (input) and mapping it to be labeled either cat or dog.

  Regression allows us to predict a continuous outcome variable based on predictor variables. Generally, regression problems deal with real numbers. For example, trying to predict home prices based on recent sales in an area.

  Evaluation metrics explain the performance of a model and can provide insight to areas that might need improvement.

  In our example, we built an MLAI model to predict a color. Everytime the model chooses a color, a score will be added to their box. Actual values are on the y-axis and the model’s predicted values are on the x-axis.

  Accuracy

  Accuracy is the total number of correct predictions divided by the total number of predictions made by the model. Accuracy works well on balanced data: a dataset with approximately the same number values across each category present. For example, 100 dogs and 95 cats is a balanced dataset. An unbalanced dataset is 100 dogs and 11 cats.

  

  18 (correct predictions) / 22 (total number of predictions) = 0.81
  Accuracy = 81%

  

  15 (correct predictions) / 22 (total number of predictions) = 0.68
  Accuracy = 68%

  Classification accuracy can give a false sense of achieving high performance. Such can occur when there is a disproportionate number of 1 color over the other (e.g. 100 blues and only 11 reds).

  Models can be evaluated based on four features:

  1. True positives are the correctly identified predictions for each class.

  2. True negatives are the correctly rejected predictions for each class.

  3. False positives are incorrectly identified predictions for a certain class.

  4. False negatives are incorrectly rejected data for a certain class.

  We will use these features to identify how well model 1 did at identifying blue.

  • The true positive rate for model 1 predicting blue was 10 (correct) / 13 (overall predictions)

  • The true negative rate for model 1 predicting blue was 8 (correctly rejected) / 9 (total rejected)

  • The false positive rate for model 1 predicting blue was 1 (incorrectly identified) / 11 (total predictions)

  • The false negative rate for model 1 predicting blue was 3 (incorrectly rejected) / 11 (total rejected)

  Precision

  Precision is the number of correct positive results divided by the number of positive results predicted by the ML model. Using precision, we discover the amount of positive identifications that were actually correct. Precision is helpful for understanding how good a model is at predicting a specific category. This metric is helpful in multi-category classification problems. For example, predicting which color car to sell.

  Precision = True positive / True positives + False positive

  

  Referring back to our model 1 color identifier from above, the true positive value would be 10 and the false positive value would be 3.

  Precision = 10 / (10 + 3) = 0.769
  Accuracy = 77%

  Recall

  Recall is the number of all positive results divided by the number of all relevant samples. Recall answers the question: what proportion of actual positives were identified correctly? A model that produces no false negatives has a recall of 1. Recall is useful when assessing whether a model can effectively detect the occurrence of a specific category. For example, if you need a model that can catch all fraudsters on your website then you’ll want a model that has high recall. A model with high recall may have low precision (gets a lot of predictions wrong) but it’s really good at detecting all the fraudsters.

  Recall = True positive / True positive + false negative

  Referring to our color identifier model, the true positives are again 10 and the false negative is 1.

  Recall = 10 / (10 + 1) = 0.909
  Recall = 91%

  F1 Score

  

  F1 score tries to find the balance between precision and recall. It tells how precise and robust a classifier is and ranges on a scale of [0,1]. The greater the F1 score, the better the performance of the model.

  

  Mean absolute error

  Mean absolute error measures how far predicted values are from observed values averaged over all predictions.

  In our example, our model will be based on predicted grocery item prices vs the actual grocery item prices.

  

  The error can be predicted as: actual cost — predicted cost. After subtracting the two, we change any negative number to a positive by taking the absolute value of that number.

  The number of training sets in our data is 5; therefore, we will assign the variable n to equal 5

  MAE = (absolute error A + absolute error B + absolute error C + absolute error D + absolute error E) / n

  MAE = (1 + 2 + 1 + 1 + 2) / 5 = 1.4

  We’re therefore able to identify that our model predictions were off my $1.40 on average .

  Root mean squared error

  Root mean squared error is the square of the difference between the original values and the predicted values. A result from squaring errors is that larger errors become more pronounced.

  Going off to our example above the RMSE would be as followed:

  Apple = 2 − 1 = 1, 1^2 = 1
  Pasta = 3 − 5 = -2, -2^2 = 4
  Almond Milk = 4 − 5 = -1^2 = 1
  Avocado = 1 − 2 = -1^2 = 1
  Ice cream = 3 − 5 = -2^2 = 4
  RMSE = (1 + 4 + 1 + 1 + 4) / 5
  RMSE = 2.2

  Algorithms

  

  There are many great algorithms, some are better than others depending on your use case or data availability. Here’s a useful visual diagram of when to use which algorithm:

  Here’s a brief overview of some of the more popular algorithms used below:

  Linear regression

  Regression models target values based on independent variables. Linear regression is a type of regression analysis where there’s a linear relationship between the independent and dependent variables.

  By finding the linear relationship between values, machines are better able to forecast upcoming variables.

  A simple linear regression would be the time spent studying (x-axis) to the grade received on a test (y-axis). Presumably, the most time spent studying would result in a higher grade.

  

  Logistic regression

  Tests whether an independent variable has an effect on a binary dependent variable. In ML, logistic regression is used as a classifier to execute tasks.

  Logistic regression is used for its ability to provide probabilities which can be used to classify new data.

  For example, logistic regression can be used to determine whether or not an email is spam. Spam emails will be categorized based on key words and features that show up in those emails. When new emails come in, it can determine whether or not it’s spam based on those identified characteristics.

  

  K-Nearest Neighbors

  Uses data points surrounding a target data point in order to make predictions about the target. It assumes that similar things exist in close proximity to each other.

  KNN can be used in problems that are built on identifying similarities.

  KNN can be used to identify patterns in buying behavior and enable businesses to use that to increase sales. For example, a grocery store can use KNN to identify the types of people who buy wine on Friday. The grocery store can find similar customers using this model and then tailor their marketing to that demographic.

  

  Decision trees

  We use decision trees in our everyday life when making choices. For example when buying a new car we usually decide on a price point and look for things below that point. Then we may choose the type of seats we want. Based on what is available with the type of seat we want, we choose the color car we want. Decision trees are commonly used when working with non-linear datasets.

  

  Random forest

  Creates an algorithm based on how many decision trees (creating the ‘forest’) are available. The more trees, the more accurate the algorithm’s prediction will be.

  Random forest can cross validate inputs, thus, making its output more accurate. This algorithm is used when handling large datasets.

  Stock traders can use decision forests to predict a stock’s future behavior. They do this by taking decision trees of similar stocks which cross validate each other’s input which can make predicting the potential of a new, similar stock more accurate.

  

  Support vector machines

  Hyperplanes are decision boundaries used to classify data. Support vectors are data points lying closest to the decision line that are most difficult to classify. SVMs find an optimal solution to classify these data points.

  SVMs have a high speed and high performance accuracy with a limited amount of data.

  SVMs are being used to classify cancer using image classification. The SVM algorithm runs models with datasets of thousands of cancerous cell images. It classifies the image around the cell and after processing thousands of images, can accurately classify scans in real time.

  

  Neural Network

  Neural networks are a series of algorithms used to find relationships in data.Neural networks are able to learn by themselves, thus, have the ability to produce an output not limited to the given input. This algorithm has been implemented in image object detection, language understanding, audio detection, and more.

  Neural networks have been used to teach a machine how to play chess. Analyzing over 2.5 million chess matches, this neural network predicts chess moves based on the history of moves in the game. This is significant because it demonstrates the ability of a machine to learn and understand game theory.

  

  Conclusion

  With the right foundation, developing AI/ML is an accessible goal. You’re now ready to start coding. Check out how to build your first machine learning model: https://m.mage.ai/how-to-get-started-with-ai-ml-8630fecfd776

Topics Discussed:

1. What is Airflow?

2. Our previous setup

3. Challenges running Airflow in ECS

4. Managed Airflow alternatives

5. Our current setup

6. Migrating to Astronomer Cloud

   Deploying Airflow to production using Astronomer Cloud

   

What is Airflow?

Apache Airflow is an open-source data orchestration tool for programmatically creating workflows using Python. A workflow includes directed acyclic graphs (DAGs) which have tasks connected to one another but never in a closed loop.

Why should you care?

Workflows can automate and streamline repeatable tasks, increasing productivity and efficiency for your project or business. Creating these workflows without a tool like Airflow can be a difficult and increasingly unmanageable endeavor as the number of tasks grows and the dependencies between tasks become more and more complicated.

If you need to process large amounts of data and have jobs with many steps running on a recurring schedule, Airflow can make your life a lot easier.

How we use Airflow at Mage

Airflow is an essential part of our software stack. Mage is a tool that helps developers use machine learning and their data to make predictions. We use Airflow to orchestrate the steps required in preparing training data, building new machine learning models, deploying machine learning models, processing streamed data, and more.

Airflow is an important part of making sure steps in our workflow happen in a certain order and on a recurring basis.

Options for running Airflow in production

There are many different methods for running Airflow in production, depending on the needs of your project. Though Airflow comes with SQLite by default, it will need a production database like PostgreSQL or MySQL for storing metadata. Amazon’s Relational Database Service (RDS) or Google’s Cloud SQL are two options for hosting the database.

The other components of Airflow can be hosted in different cloud providers or on your own bare-metal server. Here is a non-exhaustive list of various cloud services you can use (get ready for lots of acronyms):

Amazon Web Services

• Elastic Compute Cloud (EC2)

• Elastic Container Service (ECS)

• Elastic Kubernetes Services (EKS)

Google Cloud Platform:

• Google Compute Engine (GCE)

• Google Cloud Run

• Google Kubernetes Engine (GKE)

Microsoft Azure

• Azure Virtual Machines (VMs)

• Azure Container Instances (ACI)

• Azure Kubernetes Service (AKS)

Heroku

Our previous setup — Amazon ECS (Elastic Container Service)

For our previous setup, we used an Amazon ECS cluster running Airflow version 1.10.14. In this ECS cluster, we utilized five services: Workers, Webserver, Scheduler, Flower, and the Celery Executor.

In addition, we had a Redis cluster in Amazon ElastiCache and a PostgreSQL database hosted in Amazon RDS. Refer to the Appendix at the bottom of this post for more details on the Airflow components.

Challenges running Airflow in ECS

As the number of DAGs in our Airflow cluster grew, we noticed that DAGs would sometimes have issues running successfully, or there would be problems with the scheduler. We also had unnecessary costs due to over-provisioned Airflow workers, and it became a hassle to find the right resource allocation through the Amazon ECS UI.

Managing the number of Airflow workers and resources allocated to the various AWS services required for Airflow was a bit cumbersome. We needed a way to easily manage these resources and quickly deploy changes to our infrastructure if necessary. We wanted to be able to scale up Airflow task processing efficiently, effectively, and with as little overhead and maintenance from the team as possible, and we were willing to pay for a managed service to help us with that.

Managed Airflow alternatives

Amazon Managed Workflows for Apache Airflow (MWAA)

MWAA is a managed service in AWS for Apache Airflow. It’s backed by Amazon and meant to integrate with other Amazon services. Google Cloud has their version of a managed service for Airflow in Cloud Composer, but since Mage’s cloud infrastructure is built in AWS (future article on this coming soon), we won’t be mentioning the advantages or disadvantages of using Google Cloud Composer as it wasn’t an option for us.

Pricing for MWAA or Astronomer Cloud isn’t mentioned below because the difference in pricing between the two wasn’t so drastically different that it was a deciding factor in choosing one over the other.

Pros

Autoscaling workers: MWAA is a fully-managed version of Airflow, so it makes it easier to scale up Airflow. If more worker instances are needed to process tasks, MWAA takes care of it automatically and removes workers when they are no longer needed.

Compatibility with other AWS services: MWAA is part the AWS ecosystem and connects well with the tools and other resources required for your workflow if you are already hosting your infrastructure on AWS. For example, it has logging already integrated with Amazon CloudWatch.

Cons

Outdated Airflow Version: At the time of writing (April 2021), the latest Airflow version available in MWAA was 1.10.12. The current latest version of Airflow is 2.0.2. If you wanted to use Airflow 2.0, that’s automatically a dealbreaker for integrating MWAA as your tool for managing Airflow. Though MWAA will inevitably support Airflow 2.0, this may or may not mean that MWAA is slower to adopt the latest Airflow versions.

Required VPC infrastructure: If you require or already have a specific Virtual Private Cloud (VPC) infrastructure in AWS, creating a new MWAA environment may require adjustments to your current VPC or an entirely new VPC. This may not be an issue for you, but it can be a hassle if you just wanted to add the MWAA environment into an existing VPC but weren’t able to due to certain restrictions.

DAGs have to be uploaded to S3: Amazon Simple Storage Service (S3) is used to read all DAGs in MWAA. This could be fine for some workflows, but it could also result in making certain issues harder to debug when new DAGs are deployed. If you want flexibility in where you manage your DAGs and don’t want to go through the added step of uploading them to S3, this requirement for using MWAA may not be worth the additional effort, especially for existing Airflow projects.

Astronomer

Astronomer is a commercial managed Airflow platform that is cloud-agnostic. It can be used with your own cloud environment, whether it is AWS, GCP, or another cloud provider.

Pros

DAGs maintained in Git repositories: You can maintain the folder structure of your current Git repo and dynamically build DAGs without issues potentially caused by importing DAG files to an Amazon S3 bucket. Astronomer allows you to deploy DAGs directly from Git repos.

Cloud-agnostic: Astronomer is not confined to one cloud provider. You can also deploy your Airflow project to Astronomer’s own cloud infrastructure.

Easy initial setup: Astronomer Cloud setup and configuration is fairly straightforward following their documentation and doesn’t require additional security configuration regarding access keys or VPCs, since the Airflow project is hosted in Astronomer’s VPC. However, you can configure your own security settings if you use Astronomer Enterprise.

Customer support: Astronomer users have access to a customer support portal in which they can get quick responses to issues (typically within a business day or two or faster if needed, though it will cost extra) and help with troubleshooting Airflow issues. Astronomer’s business is built around Airflow so their customer support is specialized in Airflow.

CI/CD integration: Airflow deployments through Astronomer can be integrated into CI/CD pipelines using several different CI/CD tools like CircleCI, Jenkins, etc.

Quick deployments: Astronomer’s UI allows you to adjust resources allocated to the scheduler, webserver, and workers all on one page and deploy the changes within a few minutes. Not having to go through several screens to make these changes is convenient if you’re using Astronomer Cloud.

Cons

Limited logs visibility: There may be issues viewing the worker logs, as well as historical scheduler or webserver logs, through the Astronomer UI in Astronomer Cloud.

Log retention period: The log retention period for Astronomer Cloud is 15 days, so if you want access to logs older than that, you may not be able to view them through the Astronomer UI. If you have remote logging enabled, you can still have access to task logs older than 15 days.

Paying for idle resources with the Celery executor: If you’re using the Celery executor and have a lot of resources allocated to your workers but only use your workers for a brief period during the day, you still have to pay for the full capacity of the workers even when they’re not being fully utilized. You do have the option to use the Kubernetes executor, though.

Astronomer currently has two offerings: Cloud and Enterprise.

Astronomer Cloud:

Astronomer Cloud manages all of the infrastructure required to run Airflow. Your Airflow project is run on a Kubernetes cluster and hosted in Astronomer’s Google Cloud environment.

Astronomer Enterprise:

Astronomer Enterprise allows you to run and manage Airflow in your own cloud environment while still utilizing features and support from Astronomer. The enterprise version can be installed on the following Kubernetes environments: AWS Elastic Kubernetes Services (EKS), GCP Google Kubernetes Engine (GKE), or Microsoft Azure Kubernetes Service (AKS).

Astronomer Enterprise gives you more control over your infrastructure hosting Airflow, so you can customize your security settings, such as IAM roles, authentication, and VPC configuration. In addition, you have access to Grafana (a monitoring tool for Airflow metrics), Kibana (a visualization tool for your Airflow logs), and other tools.

Our current setup — Astronomer Cloud

We ultimately ended up going with Astronomer Cloud as our managed service for hosting our Airflow cluster. We did try setting up our Airflow cluster in Amazon MWAA initially because all of our infrastructure was already using AWS, so we thought the transition would be easier.

However, we quickly ran into issues configuring the MWAA environment because we had to create a separate VPC from what our other AWS services were hosted in. Our Airflow setup at the time required a connection to our Redis cluster in Amazon ElastiCache, but we could not make the connection because the Redis cluster was in a different VPC, so we likely needed to create a VPC peering connection. We also had issues building all of our DAGs because we have many dynamically-generated DAGs, and the MWAA workflow requiring DAG files to be uploaded to S3 would have required a lot of big changes to our code.

As for Astronomer Cloud versus Astronomer Enterprise, Astronomer Cloud made more sense for us because we wanted a solution that managed most, if not all, of the infrastructure required for Airflow for us. We had to make some adjustments as mentioned later in this post, but we wouldn’t consider them major changes. As our company grows, Astronomer Enterprise is still an option, and it’s common for companies to start off using Astronomer Cloud and then later switch to Astronomer Enterprise as their needs change.

Migrating to Astronomer Cloud

Upgrading to Airflow 2.0

If you’re not already using Airflow 2.0, you can follow Airflow’s upgrade guide in their documentation to upgrade.

We were using Airflow 1.10.14 and only had to make some minor changes in order to use Airflow 2.0. These are the changes we made:

1. Replaced all usages of the deprecated airflow.operators.python_operatorpackage with theairflow.operators.pythonpackage.

2. Removed the provide_contextargument when calling the PythonOperator since that parameter is deprecated and no longer required in Airflow 2.0.

3. Updated environment variables related to remote logging. We had remote logging enabled in Amazon S3, so we had to update these environment variables:

i. AIRFLOW__CORE__REMOTE_LOGGING —> AIRFLOW__LOGGING__REMOTE_LOGGING

ii. AIRFLOW__CORE__REMOTE_BASE_LOG_FOLDER —> AIRFLOW__LOGGING__BASE_LOG_FOLDER

iii. AIRFLOW__CORE__REMOTE_LOG_CONN_ID —> AIRFLOW__LOGGING__REMOTE_LOG_CONN_ID

iv. AIRFLOW__CORE__LOGGING_LEVEL —> AIRFLOW__LOGGING__LOGGING_LEVEL

There are other Airflow environment variables that were deprecated in version 2.0, so you may want to check the docs to see if your existing project is using any of them.

Removing and updating files

We were building our own custom Docker image for setting up the various Airflow services using a Dockerfile with the base image puckel/docker-airflow:1.10.9. This Dockerfile had an ENTRYPOINT that pointed to an entrypoint.sh script which defined several environment variables and installed Python packages.

Since Astronomer manages their own Docker images with all the Airflow configuration for the webserver, scheduler, and Postgres database out-of-the-box, we were able to delete the following files from our root Airflow directory:

• entrypoint.sh

• airflow.cfg

• docker-compose.yml

We no longer needed all the instructions in the Dockerfile, so we deleted everything inside it and simply replaced it with one command:

FROM quay.io/astronomer/ap-airflow:2.0.0-4-buster-onbuild

You can check Astronomer’s Docker container registry at quay.io to find the latest Airflow Docker image. One of the packages in our requirements.txt file was apache-airflow==1.10.14, but we removed that since Astronomer's Docker images already have Airflow installed.

We had used a Redis cluster in Amazon ElastiCache to dynamically build DAGs, so an Airflow connection to Redis was required. This worked well for us at the time because our Airflow services and Redis cluster were all hosted on the same Virtual Private Cloud (VPC) in AWS.

Since production Airflow projects are managed in Astronomer’s own cloud environment for Astronomer Cloud, there were issues with Airflow connecting to our ElastiCache Redis cluster. Instead of trying to create a VPN connection to our VPC containing the Redis cluster, we took advantage of the built-in Airflow Variables as it also acts as a key-value store. This required refactoring all the areas of our code where we were setting Redis keys and getting Redis values.

from airflow.models import Variable
from datetime import datetime, timedelta

import json

class AirflowVariable():    
def get_json_variable(self, key): 
       return json.loads(Variable.get( 
           key,
          default_var='{}'),
        ).get('value', None)

        def set_json_variable(
        self,
        key,
        value,
        expires_after=timedelta(hours=1),
    ):
        Variable.set(
            key,
            json.dumps({
                'value': value,
                'expires_at': (
                    datetime.utcnow() + expires_after,
                ).isoformat(),
            }),
        )

An example of Airflow variable helper methods for storing values as json

Installing and using Astronomer locally

• Follow the steps in Astronomer’s CLI Quickstart Guide to install and start Astronomer locally, taking note of the items below.

• When initializing an Airflow project on Astronomer, enter your existing Airflow root directory where the dags folder is located in. You don't need to create a new directory and cd into it, since you already have an existing Airflow project.

airflow # root directory
├── dags
├── Dockerfile
├── README.md
└── requirements.txt

A simple Airflow project folder structure

• Run the following command: astro dev init

• Some new files and folders will be generated automatically. We didn’t need to include additional files, plugins, OS-level packages, or Airflow connections, so our include and plugins folders were empty and the packages.txt file blank.

• We used to have a Redis connection as described earlier in this post, but since it was no longer needed, our airflow_settings.yaml file was also blank.

• If you previously had Docker containers running Airflow locally before installing Astronomer, make sure you stop those containers or change their ports in the .astro/config.yml file before running Airflow locally with Astronomer using the astro dev start command. There is a note with more details in the "Start Airflow Locally" step of the Astronomer Quickstart Guide.

• Earlier in this post, we mentioned that one of the files we deleted was the docker-compose.yml file. We had used it to build and deploy Docker images for the Airflow webserver, scheduler, and worker locally, but with Astronomer, it conveniently takes care of this for us without the need for a docker-compose.yml file. We just need to specify the Docker image (and Airflow version) that we want to use from Astronomer's container registry inside of the Dockerfile.

Deploying Airflow to production using Astronomer Cloud

• You can deploy your Airflow project to production by creating an Astronomer workspace and following the steps in Astronomer’s deployment guide.

• Astronomer lets you choose between three executors to execute the tasks within your DAGs. These are the Kubernetes, Celery, and Local executors. You won’t be able to change the executor through the AIRFLOW__CORE__EXECUTOR environment variable because Astronomer sets it automatically.

• At Mage, we use the Celery executor, and it has worked well for us so far, though we would consider switching to the Kubernetes executor in the future if the situation called for it.

Astronomer production deployment settings

• The screenshot above shows how much resources we allocated to our Celery workers, the webserver, and the scheduler. This setup costs $670/month, but you can see the monthly cost for other resource allocations when you adjust the sliders for the settings.

• This required some experimentation to see what made sense for us while trying to avoid over-provisioning resources and spending money we didn’t need to. For context, we have around 300 DAGs that get run daily at the time of writing.

• The webserver and scheduler are at their default levels, so we didn’t have to increase resources for those yet.

Environment variables to take into account for scalability

• Since we removed the airflow.cfgfile and no longer need it when using Astronomer, we make changes to environment variables (the different methods are explained in Astronomer's documentation) if we want to update the Airflow configuration.

• AIRFLOW__CORE__PARALLELISM - Max number of overall task instances to run in parallel; default of 32.

• AIRFLOW__CORE__DAG_CONCURRENCY - Number of task instances to run concurrently in one DAG; default of 16.

• AIRFLOW__CELERY__WORKER_CONCURRENCY - Number of tasks each Celery worker can process at a time; default of 16. We initially set this way too high given the limited resources allocated to one worker, so be careful not to set an arbitrarily high number or your DAGs may not get properly executed. We have this value set at 30 in our current setup.

• AIRFLOW__CORE__DAGBAG_IMPORT_TIMEOUT - Seconds before timing out a Python file import; default of 30. If you have dynamic DAGs that may take some time to parse, increase this value. We have it set to 3600.

• Depending on how many DAGs you have, the number of Airflow workers, and the amount of resources allocated to each worker, you’ll need to adjust the environment variables above to accommodate your Airflow configuration. Refer to Airflow’s configuration reference for all the other environment variables you can use, as well as Astronomer’s guide on scaling out Airflow for more details.

Closing thoughts

Our Airflow setup and configuration isn’t perfect and we’ll inevitably have to make adjustments as our company grows. But if you’re planning on moving your Airflow project to a managed solution, hopefully this post saved you the headache of dealing with some of the issues we ran into and may serve as a starting point. If you have any questions related to this post or about our company or just found some errors you wanna tell us about, feel free to email eng@mage.ai. We are always open to all types of feedback and looking for ways to improve our content!

Airflow Components Appendix

Workers: Workers function to execute the logic of tasks. By listening to message queues, workers pull the desired tasks of a DAG and process them.

Webserver: The Webserver UI (user interface) of Airflow provides the ability to control and visualize aspects of each pipeline. Two important functions that the Webserver provides are monitoring DAGs and providing DAG visualization tables. This shows each DAG’s runs and the status of every task’s runs.

Scheduler: The Airflow scheduler is a Python process which checks the code in the Airflow DAG folder. The scheduler manages DAG runs, tasks, and the storing of DAGs. From there, it decides which DAGs should run on each pool and will monitor the DAGs to see which are running successfully and which need to be executed or retired.

Flower:Flower is a tool which monitors and administers Celery clusters — distributed task queues based on distributed message passing.

Celery Executor: The executor is responsible for running each task. There are various types of executors, but we used the Celery executor as it allowed for horizontal scaling of workers without the need for Kubernetes. Astronomer has a great guide in their documentation explaining Airflow executors.

Redis: We used Redis (an in-memory data store) as the Celery broker for message transports. We also used Redis to dynamically build DAGs by regularly making calls to our backend to retrieve live model data and storing it in Redis. The DAGs would get updated or created based off of this data in Redis.

PostgreSQL: This was our metadata database used to store configurations, variables, tasks, schedule intervals, and other relevant data.

To help other startups get ahead, we are sharing what software we currently use at Mage (tech related SaaS such as AWS, Datadog, Effx, etc. will be shared in a future article).

Here are the categories of SaaS we cover in this edition:

1. Coding

2. Design

3. IT/Security

4. Communication

5. Documentation

6. Project management

7. CRM

8. HR & payroll

9. Legal services

10. Finance & accounting

11. Stocks & compensation

Coding

Using the right text editor software can drastically increase productivity by catching errors in your code, increasing automation, keyboard shortcuts, and more.

What we use: VSCode, Sublime, PyCharm

Alternatives: Atom, Eclipse

We use Sublime as one of our text editors. Drawn to it for its “fuzzy search” which allows for a quick recovery of files based on a few letters or words, Sublime integrates well with our functions. Another feature we utilize is the package manager which allows us to install community created modules that add functionality and a customized look and feel. Sublime is also more lightweight than other software and thus loads much faster.

Another staple coding tool we use is VSCode. VSCode has several out-of-the-box features that set it apart from other tools: searching code across several directories, built-in git features, plethora of extensions, and integrated terminals. It provides a great interface and shortcuts that are familiar to our programmers. A huge pro of VSCode is that its open source; meaning anyone can add new improvements to software that benefits every user.

PyCharm’s IDEs (integrated development environments) are extremely beneficial to the developer: superfast code navigation and indexing, automatic code linting and reformatting, intuitive git integration, powerful refactoring capabilities and import optimization, great error catching, and support for third party plugins. PyCharm’s IDE works well in tandem with Python’s type hints as it unlocks a new layer of error catching and information to present. When using a debugger, PyCharm is powerful, a joy to use, and has helped to avoid countless difficult to track bugs and problems.

Design

Delivering magical experiences (one of our core virtues) to customers through an intuitive and simple product should be the main objective of any product team. This is done in large part by using the right design tool. We believe that collaboration produces the best result (that’s why Mage is a collaborative AI tool for developers), and thus sought out a design tool that can deliver that need.

What we use: Figma

Alternatives: Sketch, Affinity Designer

For our design processes, we use the cloud-based design tool Figma. We use Figma to draw low-fidelity wireframes and product flows in the early stages of ideation. Towards the later stages, we do high-fidelity screen designs, prototyping, and lots of pixel pushing. At every stage of the process, we get feedback from our stakeholders. Once our designs are agreed upon, our developers can get the specs they need. This streamlined workflow saves time and effort for every person involved in the process. With Figma, there are no downloads needed; making it easy to share with anyone involved. Using Figma in this way is a great way to include non-design partners in the design process.

IT/Security

Cloud-based IT software can be used for password protection, data security, and compliance solutions. Integrating a SaaS security solution into a startup increases IT’s efficiency by providing up-to-date software to protect against cybersecurity threats.

Password manager

What we use: Last Password, 1Password

Alternatives: aWallet, Bitwarden

Last Password is the number one password manager tool and works well for our needs. By logging into your accounts on their software, Last Password will secure your passwords in one safe and easy to access platform. Last Password also generates strong passwords, has a sharing feature, and alerts customers when personal information is at risk. 1Password uses similar features and technology that keep passwords safe while allowing team members access to these passwords.

Single sign-on

What we use: Okta

Alternatives: OneLogin

Okta’s cloud-based software helps companies guarantee proper user authentication when logging into accounts. It also allows developers to build identity controls into applications and web services. Okta’s interface is easy to navigate, and its popularity makes it familiar to most team members and customers.

2FA (aka two factor authentication)

What we use: Google Authenticator

Alternatives: Duo

Google Authenticator is an easy-to-use, straightforward two-step verification service.

Mobile Device Management

What we use: Jamf, Apple Business Manager, Apple Automated Device Enrollment

Alternatives: Hexnode, Kandji

Jamf provides a free and straightforward setup to effectively manage all Apple devices. We use this in tandem with Apple Business Manager and Apple’s automated device enrollment program.

Compliance

What we use: SecureFrame, VGS, Kolide

Alternatives: CyberOne

Kolide can help to communicate your security guidelines to employees, integrated with Slack, and recommends clear and easy to resolve action items.

SOC 2 is an auditing process designed to protect the privacy of an organization and clients. It’s an eventual requirement for SaaS companies who store customer data. Using software can help your company to navigate the important information and requirements needed for a successful audit.

For our SOC 2 process, we are choosing between SecureFrame and VGS. VGS’s security foundations are free and helpful in getting started with the SOC 2 process. Secureframe offers great customer support to get our certification complete from end-to-end. They also provide a lot of automation during the process to reduce the amount of time our developers need to spend working on SOC 2 compliance.

Communication

With most teams working remotely, it becomes essential for businesses to implement communication software that can help facilitate the brainstorming and the exchange of ideas that comes naturally with spontaneous workplace communication.

Internal Collaboration

What we use: Slack, Tandem, Vowel

Alternatives: Google Hangouts, Microsoft Teams

We use Slack for its depth of features: file sharing, messaging, and customer integration which provides our team with an easy-to-use, well-managed communication software that is the solution to long email chains. Slack integrates well with other apps. Our team pairs Slack with GitHub which increases productivity by displaying our development work alongside the team’s communication channel.

For voice calls, small video chats, and real time collaboration, we found Tandem to be the best on the market. Its desktop integration allows teams to talk and collaborate without links and meeting rooms. It also allows the team to share what they’re working on and when they’re available to talk and collaborate. We found this to be the closest thing to working in an office.

Finally, we use Vowel for our large team meetings. Vowel not only provides video meetings, but a meeting management service. Its features and integrations: automatic transcription, assigning action items, and setting agendas allow for a much more productive and efficient way of running meetings and recalling discussion notes.

External Communication

What we use: Zoom

Alternatives: Google Meet, UberConference, Skype

We found Zoom to be the best service to hold customer meetings. Zoom’s superior video quality and audio was reason enough for us to choose it over other video conference software such as Google Meet or Skype. The depth of Zoom exclusive features: recording, waiting room, and participant interaction allows us to conduct the most professional and productive meetings possible with our customers. Zoom isn’t perfect though, it has many flaws such as complicated default settings such as not allowing other participants to share their screen without changing a setting in Zoom.

Email

What we use: Gmail

Alternatives: Outlook, Spark, Superhuman

Having quick email communication is one of the easiest ways to boost customer support satisfaction. Using a software that can help organize your inbox and track conversations with teammates, customers, and investors will result in better team organization, communication, and efficiency.

For its widespread availability, easy to use interface, and multitude of features, we use Gmail for our email correspondence. Gmail makes sorting your inbox extremely easy to do by creating categories based on keywords or senders. The cloud-based software allows for email access on any device and nearly unlimited free storage. Gmail’s streamline interface and wide range of features integrated best into our company.

Documentation

Choosing the right document management software is key to streamlining workflows, scaling the team, and properly organizing business materials. We sought out a document sharing software that could foster collaboration amongst team-members.

What we use: Notion, Google Docs

Alternatives: Slab, Coda, OneDrive

We are currently using a combination of Google Docs and Notion. Google Doc’s straight-forward, easy to use approach to editing and collaborating on content is top-notch. One downside to Google Docs is that it’s hard to find documents. We use Notion for its ease in browsing and discovering documents that you may not know existed. While we are still testing out Notion, we found the app difficult to copy and paste large sections and to tab list type sections on mobile devices; such as numbered or bullet lists.

Project management

Effectively managing projects and their processes aids in shipping valuable features to your customers. Using a project management software can lead to better team collaboration to increase the quality of the product and decrease time it takes to create it.

What we use: Airtable

Alternatives: Linear, Google Sheets, Asana, Monday, Jira, Trello, Wrike

Airtable is a necessary part of our weekly business operations. As straightforward as a Google Sheet, Airtable goes further by offering a better interface, flexible database, and many more features: ability to create multiple tables, sorting different types of data, linking records together. Its integration with Kanban boards and storage in different columns provides helpful information when tracking projects. Read more on how we use Airtable by reading “How we organize our software development process”.

CRM (customer relationship management)

Productively using a CRM (customer relationship management) tool can give your company insight on customer behavior and information that will aid in creating and adapting products best suited to customer needs.

What we use: Clay

Alternatives: Salesforce, Hubspot

We use Clay because it’s very flexible and we can configure it to our specific needs. It’s as flexible as a Google Sheets but has a superior user experience. One of the biggest pros of using Clay is how programmable it is. You can utilize its vast capabilities right out of the box. However, they provide many no-code and low-code functions that allow you to fetch data from various sources and map that data to your columns. Furthermore, you can have different functions’ output be the input into another function. For example, we have a function that looks up a company website URL based on the company name. Then, using the inferred website URL, we have another function to look up more details about that company which are derived from their website and other related sources.

Community management

It’s through the use of a community management tool that we are able to understand the needs of our customers and build the most useful products. Without the right community management software, you will fall out of touch with your users and build the wrong product.

What we use: Discord, Yen

Alternatives: Orbit, Open Social

We use Discord as our primary community management software. Discord servers can accommodate hundreds of users in one company’s channel at no additional cost to the company. We can use this service to host webinars and tutorials on our software to better grow our brand and build up our community.

While not currently available for public access, we plan to use Yen as soon as it launches out of stealth. Yen allows for potential users, team members, and your community a way to engage with one another without leaving your website. This integration allows for a better user experience as no additional app needs to be downloaded and all community feedback, engagement, and reviews can be easily accessed through your website.

HR & payroll

Equipping your HR team with the right tools for success is essential in hiring the right talent and implementing practices that will make your startup run smoothly. Using a SaaS-based HR program allows businesses the power to manage all the classic HR functions and use them to evolve with advancing technology.

What we use: Gusto

Alternatives: Rippling, Quickbooks

Gusto is our preferred choice as it offers the most all-encompassing functions: manage payroll, benefits, new team member on-boarding, project tracking, team directory and surveys, company policies, and more. The platform is user-friendly and comes with a very helpful and responsive customer service team, unlike other platforms we tested. Gusto had the most transparent pricing with no additional or hidden fees and since we used Brex for our banking, we received 50% off for the first 12 months of Gusto.

Legal services

With any startup comes lots of legal paperwork and documentation. Learning how to navigate, read, and write such paperwork can be extremely time-consuming and riddled with errors. We use legal assistance software to increase efficiency, saving ourselves time and money.

What we use: Clerky

Alternatives: Rocket Lawyer, LegalShield

We found Clerky to be the best fit for our functions. Their interface is easy to navigate and read even for those not equipped with a legal background. Clerky’s team provides responsive customer support who will answer your questions and flag paperwork issues. The platform offers legal assistance for formation, fundraising, and hiring.

Note: Clerky is great to get started quickly. However, if you start raising money and your company grows, you’ll want a team of great corporate lawyers by your side. We use Cooley because they have very smart people who feel like part of our team.

Finance & accounting

Banking

SaaS financial services help to keep your cash safe, monitor and pay bills, and accumulate corporate credit card rewards. For online platforms, it’d be very difficult to facilitate any of these transactions without the proper software.

What we use: Brex, SVB, NorthOne

Alternatives: Mercury, Rho

For our financials we use a combination of Brex and SVB. Brex offers an easy to use mobile app and website interface making it easy to access and view funds. Their platform is very affordable for startups as it offers free transactions, no wire fees, and no minimum balance requirement.

SVB’s customer service is top of the line and their vast experience working with startups has made them experts in startup banking. Banking with SVB is ideal for startups because of its three-year period of no/low fees, unlimited transactions, as well as access to networking events and webinars. Putting money away with SVB gives customers a 1% interest rate. SVB falls short in that their online interface feels dated and functions like opening an account, can be confusing to navigate and could take a few days to complete.

We just learned about NorthOne and applied to become a member. Their automatic categorization of purchases, reporting, team sub-accounts, mobile app, instant customer support, and focus on smaller companies is a huge need for us.

Credit Cards

What we use: Ramp

Alternatives: American Express Business Platinum, Chase Business

For credit card purchases, we use Ramp. Ramp offers 1.5% cash back on all purchases, and its intuitive software keeps spending reports well organized. This helps us better monitor our spending and more easily create reports for financial planning and analysis.

Accounting

Accounting services are used to track financial statements, monitor and reconcile bank accounts, facilitate purchase orders, and cash flow. Keeping an organized track record of these reports is essential to the business process.

What we use: Xero

Alternatives: Quickbooks, Freshbooks

While Quickbooks is the most popular software on the market, we found Xero to be best fit for our needs. Xero’s user-friendly interface makes it easy to pay bills, manage inventory, and account for assets. While Xero lacks a payroll service, it integrates well with Gusto. At a more affordable price point, Xero doesn’t sacrifice any of the features its competitors offer.

Stocks & compensation

Equity Management/Stocks

Going at equity management alone by using spreadsheets is error-prone and can hold back the daily functioning and success of a business. Equity management software allows startups a medium to manage their cap tables, valuations, and equity plans which can then be shared with investors and the team.

What we use: Carta

Alternatives: Ledgy, Certent

Carta offers us an equity management tool that simplifies the way we manage our shares. Carta’s well constructed user interface allows for non-experts to read, construct, and share cap tables in an intuitive and informative way. Carta beats out its competitors by offering an extensive list of features which cover all the equity management assistance a startup needs.

Compensation

To attract and onboard new talent, a well-thought-out compensation package is needed to create and reflect a new hire’s experience as well as the size of company, industry, and location. Using SaaS tools to create compensation packages can create the most accurate package, while saving your company time and money.

What we use: Pequity, Levels, data from investors

Alternatives: Glassdoor, Indeed

We use Pequity because it acts as your in-house talent organization. The software helps create salaries that fit the needs of the company at every stage of growth. Additionally, they provide us a tool for equity compensation for each level and function; they even break it down by geography (helpful for remote companies). With a great user experience and responsive customer support, we highly recommend Pequity.

As additional tools, we use Levels.fyi and leverage our investors’ data and their equity packages to determine the most beneficial compensation packages to offer our top talent.

Tech stack coming soon

In a follow-up article next month (May 2021), we’ll be sharing what SaaS tools we use in our tech stack. Stay tuned!

Image augmentation is a common technique used when training computer vision models in order to generate artificial training data by transforming in your actual training data, for example, random rotations and shifts.

However, bugs are tricky because no errors will be raised. Instead the result would be that the model will not perform as well on the un-augmented test dataset as it could have.

You can use this tool to develop and sanity check your transforms on actual images before using them in a training script.

• We used Scrum at our previous startup and at Airbnb, with a few tweaks.

• We currently use a hybrid of Scrum and Kanban at Mage (AI application development tool), with a few tweaks.

• Craft your process to maximize your team’s autonomy, collaboration, and adaptability; thus, increasing your team’s effectiveness.

What is it?

According to Wikipedia, “a software development process is the process of dividing software development work into distinct phases to improve design, product management, and project management”.

Why should you care?

A good process helps your team be more effective in delivering value to customers by empowering your team to be autonomous (make decisions on their own), collaborative (sum is greater than the parts), and adaptive (ship relevant products that customers need).

Our process in the past

At OnMyBlock (our previous startup) and at Airbnb, we used Scrum (with a few tweaks) as our software development process.

Every 2 weeks, our team would get together to review all the work that was done in the previous sprint (a time-boxed window usually 2-4 weeks long) and then plan for the upcoming sprint. This meeting is typically called sprint review and sprint planning.

Here is how we conducted the sprint review and planning:

1. Before the meeting, the scrum master (a single person responsible for project managing the stories during a sprint) would ping everyone to make sure they updated all their stories (aka tasks) in our project management software.

2. At the start of the meeting, each person goes around and shares what they shipped, what work they couldn’t finish, and explain why they couldn’t accomplish the unfinished work.

3. Then, the scrum master leads the team in discussion on which upcoming epics (aka features) are most urgent and important.

4. Everyone independently plans what they will deliver by the end of the next sprint, choosing from a list of prioritized epics. Each person is allotted a fixed number of story points. If you are allotted 13 story points (we used Fibonacci numbers to scale our story points), you can only commit to stories that have a total sum of 13 story points. Story points are values assigned to a story (aka tasks) that attempt to provide a sense of how difficult it will be to accomplish that story.

5. After everyone is done planning, each person goes around and shares what they are committing to delivering in the upcoming sprint.

There are usually “stand-ups” in Scrum, which are short daily meetings for the team to update each other. Our team felt daily stand-ups were too frequent and unproductive so we cut it down to 3 per week except we have 2 per week on the weeks where we have sprint review and planning. Our stand-ups follow the typical agenda where each person goes around and shares what they did yesterday, what they are working on today, and if they have any blockers to their current story (aka task).

Pros of how we did Scrum

• Engineers had more ownership of the features they were responsible for and this resulted in better performance reviews because we were assessed on which projects we led and delivered end-to-end.

• There was clear accountability because if someone commits to work that they didn’t finish, they would be responsible for explaining why they couldn’t finish it by the deadline.

• Less knowledge was required to accomplish tasks because an engineer can fill up their sprint with tasks that were related to their area of expertise; such as backend, frontend, data, ML, different services, specific languages, etc.

Cons of how we did Scrum

• Estimating the difficulty of stories was rarely accurate, which led to frequent over estimating and underestimating of the work.

• When we pre-assign ourselves stories, we typically chose stories that belonged to an epic we were responsible for. Many of us ended up working in silos, which decreased collaboration during the implementation phase of projects (we had a lot of collaboration during the design phase).

• Entire features shipped slower because the concentration of effort was spread out across multiple epics at once.

• It was difficult for our team to quickly adapt and change course mid sprint since we planned 2 weeks in advance.

The process today

At our current company, Mage (AI application development tool), we use a hybrid of Scrum and Kanban with a few tweaks. The tool we use to manage this is called Airtable.

Once a week, we hold a meeting called “Mage Force” where we review all the features we shipped last week and then prioritize the features we need to deliver in the upcoming release. Before we start the meeting, we hold a trust activity where the goal is to build trust amongst ourselves and to build each other up (one of our core virtues of “give people power-ups”). In contrast to having a 2 week sprint, we have 1 week release cycles where we ship as much product as possible that our customers need.

Here is how we plan our release cycle:

• We make sure that every epic (aka features) is attributed to a key result or a specific customer use case. That way, we are making sure to deliver as much value to our customers as possible.

• Each epic is given a deadline ship date that corresponds to when a customer will use it.

• Each epic is broken down into stories (aka tasks), which can be worked on in parallel

• The order in which stories are worked on is based on the related epic’s deadline and whether or not there are downstream dependent stories

• A story in the same epic with a lower priority means it depends on another story within the same epic to be completed first

• Each person who creates a story during planning will fill in as much detail in the notes to give any future engineer context on why the story exists, the requirements of the story, and any additional knowledge that will be helpful in completing the story.

• When we decide which epic must be shipped for the next release, we change all the stories for that epic to “Not started”; which will then make it appear on our Kanban board.

• Stories are not pre-assigned. When someone has no stories “In progress”, they will take the highest priority story that has “Not started”, move it to “In progress”, and assign it to themselves.

• Each story moves through the following stages on the Kanban board: Not started, In progress, Review (e.g. review pull request), QA (merged into master and need to manually test on staging), Complete (new change is on production).

• Once a story is finished, the DRI (directly responsible individual) moves it to “Review”, creates a pull request on GitHub, and associates it to the story for someone else to review.

• Once the code is reviewed, approved, and merged into the master branch, the DRI moves the story to “QA” where they must now manually test their changes on the staging environment.

• After the new changes are verified on staging, the DRI deploys the code to production and moves the story to “Complete”.

We have stand-ups (daily meetings in Scrum) 4 times a week, skipping the day we have “Mage Force”. Our stand-up meetings are called “Mission Briefings”. Each person goes around and shares what they learned yesterday and what their plan of attack is for crushing their story today. With this format, we maximize learnings (one of our core virtues of “always be leveling up”) instead of simply reporting tasks (which can be done by looking at the Kanban board).

Pros of our hybrid approach

• There is a lot more collaboration at the implementation phase because people are able to work on the same epic together at the same time.

• Working closer together is more fun and exciting, leading to higher team morale and solidarity.

• We ship new features on a daily basis, helping us make customers happier and win more new business.

• Everyone works across multiple codebases and is responsible for many areas of the product. This increases the speed in which we ship new features because everyone is capable of working across the entire tech stack, unblocking themselves if needed.

• We are able to adapt rapidly and change our focus as customer needs change or more urgent features arise unexpectedly.

Cons of our hybrid approach

• Because stories are not pre-assigned, individuals on the team are required to understand multiple code bases and technologies in order to effectively complete stories.

• There is less accountability at the individual level because each person is not committing to being responsible for delivering a set of stories. Instead, the entire team is committing to the work. If something doesn’t get done, who is responsible? We have yet to solve for this.

• Since everyone on the team works together to ship as much customer value as fast as possible, there can be a lack of “ownership” of projects. At Airbnb, engineers were awarded for having ownership of projects and therefore ownership of projects was desired by individuals. The system in place rewarded that desire and behavior.

Conclusion

Structure your software development process to maximize your team’s autonomy (make decisions on their own), collaboration (sum is greater than the parts), and adaptability (ship relevant products that customers need).

A hybrid of Scrum and Kanban, with a few tweaks, works great for Mage at this stage because we are prioritizing speed and focus through deeper collaboration.

As we grow and evolve our process, we need to keep in mind how we deal with individual “ownership” of projects and the increasing need for specialization.