Robotics: Science and Systems 2023

Daegu, Republic of Korea, July 10-July 14, 2023

Demonstrating Large-Scale Package Manipulation

via Learned Metrics of Pick Success
Shuai Li∗ , Azarakhsh Keipour∗ , Kevin Jamieson∗† , Nicolas Hudson∗ , Charles Swan∗ and Kostas Bekris∗‡
∗ Amazon Robotics, Seattle, Washington 98109, USA
† University
of Washington, Seattle, Washington 98105, USA
‡ Rutgers University, Piscataway, New Jersey 08854, USA

Email: {amzshua, keipourv, jamikevi, hudnco, cswan, bekris}@[Link]

Abstract—Automating warehouse operations can reduce logis-

tics overhead costs, ultimately driving down the final price for
consumers, increasing the speed of delivery, and enhancing the
resiliency to workforce fluctuations. The past few years have
seen increased interest in automating such repeated tasks but
mostly in controlled settings. Tasks such as picking objects
from unstructured, cluttered piles have only recently become
robust enough for large-scale deployment with minimal human
intervention.
This paper demonstrates a large-scale package manipulation
from unstructured piles in Amazon Robotics’ Robot Induction
(Robin) fleet, which utilizes a pick success predictor trained on
real production data. Specifically, the system was trained on over
394K picks. It is used for singulating up to 5 million packages
per day and has manipulated over 200 million packages during Fig. 1: A robot induction (Robin) workcell used for the
this paper’s evaluation period.
The developed learned pick quality measure ranks various pick statistics of this demonstration. The robotic arm is used for
alternatives in real-time and prioritizes the most promising ones automated package singulation by [Link], Inc. It picks
for execution. The pick success predictor aims to estimate from packages from an unstructured pile on a conveyor belt and
prior experience the success probability of a desired pick by the places them on mobile drive robots.
deployed industrial robotic arms in cluttered scenes containing
deformable and rigid objects with partially known properties. It
in another spot with the goal of singulating the items or
is a shallow machine learning model, which allows us to evaluate
which features are most important for the prediction. An online feeding them to another machine (e.g., a sortation machine).
pick ranker leverages the learned success predictor to prioritize Automation is still far from perfect, however. Some robot
the most promising picks for the robotic arm, which are then settings require a simplified environment to operate (e.g.,
assessed for collision avoidance. This learned ranking process is only a single rigid object placed around the center of the
demonstrated to overcome the limitations and outperform the
conveyor belt), while others can only deal with a subset of
performance of manually engineered and heuristic alternatives.
To the best of the authors’ knowledge, this paper presents the the target objects, and the rest are passed on to humans (e.g.,
first large-scale deployment of learned pick quality estimation apple crating). Success metrics and cost benefits vary across
methods in a real production system. tasks, and while these systems are largely beneficial, recent
advances in robotics, computer vision, and machine learning
I. I NTRODUCTION are providing additional opportunities for robots to become
Automation in the industrial, manufacturing, and warehouse financially viable in rather complex manipulation operations.
sectors has the potential to lower overhead expenses associated This work presents the learned pick quality system used in
with producing, handling, and sorting goods. The increased the Robot Induction (Robin) fleet of [Link], Inc., which
speed and precision for handling each product can lower sorts several million packages per day [1]. Figure 1 shows a
customer costs and improve product quality. Furthermore, it Robin workcell picking packages from a conveyor belt, which
can reduce risks to humans in manual operations and enhance has been used for the statistics presented in this demonstration.
resilience to fluctuations in the labor market and overall Once a package is picked, Robin scans and places the package
economy. on a mobile drive unit to be routed to an appropriate drop
Robot manipulation systems have already gained significant point.
traction across industries, from car and garment manufacturing There is variation in the arm setup among workcells due
to crating apples [16, 13, 20, 19]. Many repeated operations to real-world constraints of industrial facilities, and the exact
in industrial settings include pick-and-place tasks using robot information about the incoming items is generally unknown.
arms [17, 3]. Induction robots pick items from one location For this reason, a perception system has been developed, which
(e.g., a conveyor belt, a tote, or a box) and place them aims to provide information regarding the packages on the
conveyor belt. Even with effective perception in the loop, the learned pick quality prediction system. This strategy
however, several challenges remain for successful picking, has improved the production system’s metrics compared to
including: manually engineered, heuristic methods.
• We show that retraining the model on more recent data
• The packages have different types of material (e.g., rigid
or non-rigid, smooth or rough, etc.), requiring different improves performance, indicating that this learning system
picking strategies. For example, packages can be rigid is already effective with smaller amounts of data but can also
boxes, deformable polybags, or semi-rigid containers in a improve over time with more recent and increasing datasets.
mail sorting application. The rest of this demonstration paper is organized as follows:
• While the perception system can estimate an object’s dimen- Section II reviews the related work and state of the art;
sions and material type, the mass and mass distributions of Section III formalizes the problem considered in this work;
the incoming objects are more difficult to evaluate. Sections IV and V explain the methods for pick success
• A typical scene of Robin will contain many packages, often prediction and learned pick ranking used by the demonstrated
in a pile, with many objects only partially observed or some system; Section VI describes the evaluation performed and dis-
wholly buried in the pile. cusses the results of the corresponding tests in the production
• The fleet of robots is generally heterogeneous across work- system; finally, Section VII discusses the lessons learned and
cells. There are variations in the workcell design, the opera- future efforts in this area.
tion environments (e.g., the surface where the packages are
II. R ELATED W ORK
picked from or placed on), and the manipulator arm models,
and different end-of-arm tools (EoAT) may be used due to Having an induction scene with several objects (short-
changes in hardware over the deployment period. handed as an induct), the robot needs to execute one pick to
move an object out of the pile and place it in the desired spot
A crucial metric in the robotic induction task is the success at each step. The robot may consider many candidate picks,
rate of picking attempts. Ideally, in a pile of objects, the robot but at each stage needs to choose and execute just the one
should pick them one by one and place them in the target area with the highest predicted chance of success.
without dropping any items. Two significant types of failure Ideally, the highest-ranking candidate pick should be the
are possible: either the robot fails to find a suitable pick in one chosen for execution, eliminating the need for suggesting
the scene (e.g., due to potential collisions or if the objects are more than one candidate pick. However, a complete feasibility
somehow out of reach), or the pick attempt is unsuccessful evaluation of all potential candidates is usually impractical due
(e.g., the object is dropped after being picked). We call the to computational and time constraints. Therefore, an ordered
former as planning failure and the latter as holding failure. list of candidate picks should be provided to the robot; the
An additional type of failure arises in item singulation when robot will evaluate the candidate picks one by one and execute
inadvertently more than one item is picked and placed at the the highest-ranked candidate that passes the feasibility checks
same time. We call this type of failure multi-pick failure. (e.g., collision and robot arm reachability checks).
To deal with planning failures, in theory, it is possible Various strategies can be devised to order the candidate
to check for reachability and collisions for a large finite picks. Some of these methods are based on hand-crafted
number of picks in the scene until a viable pick is found. intuitive heuristics, such as prioritizing the larger objects or
In practice, however, computing collision-free arm trajectories the objects at the top of the pile and preferring the picks
and performing reachability checks are expensive, and the high closer to the center of the objects and the ones having a
throughput requirements of the industrial operation limits the higher number of activated suction cups during EoAT’s contact
number of picks that can be analyzed for each scene. Even if with the object. In practice, such heuristics may work for a
viable picks are found that pass the checks, there is a need to nominal induct but fail in complex scenarios and edge cases.
choose the picks that succeed in holding and transferring the Trying to manually handle all possible scenarios with more
package to the mobile robot. heuristics quickly becomes intractable. As an alternative, we
In this context, this work demonstrates how machine learn- can consider a data-driven approach that uses machine learning
ing models trained on historical pick outcomes from a pro- to learn a metric for the success probability of a particular pick
duction system can be leveraged to overcome these challenges and then rank the picks based on this score.
and improve the performance of such large-scale deployments. Pick selection based on a score learned from data has been
Moreover, as the model’s ability to predict outcomes improves an active research area in the past two decades. Morrison
with more data, the estimated pick qualities become more et al. [12] learn to estimate grasp qualities, angles, and
accurate over time. In particular, the contributions of this gripper widths for each pixel in a depth image, assuming that
demonstration can be summarized as follows: the parallel jaw gripper’s center is aligned with that pixel.
• We demonstrate a large-scale system for predicting pick The learning model is trained using a dataset of actual and
qualities using machine learning. During our evaluation, this simulated grasps. However, it does not work for non-vertical
system picked up to 5 million packages daily (i.e., over grasps and requires only a single object in its region of interest.
200 million packages over the corresponding period). Morales et al. [11] use a set of visual and geometric features
• We describe a ranking strategy for the picks, which uses with K-Nearest Neighbor clustering to predict grasp success
for BarrettHand™. However, it is only suitable for 2-D planar III. P ROBLEM S TATEMENT
shapes and does not generalize to the 3-D cluttered scene of Consider the picking task illustrated in Figure 1. The task is
industrial inducts. initiated when a scene of cluttered packages of different types
Araki et al. [2] propose a learning method for simultaneous arrives at a reachable area via a conveyor belt. The conveyor
object detection, semantic segmentation, and grasping detec- belt and the scene remain static throughout the picking process
tion. However, as a black box, it is difficult to improve the until the scene is “cleared,” which occurs when no reachable
grasp quality using this method. A more modular approach packages remain or when some exception occurs, and the next
would allow simpler debugging for settings outside the lab’s scene arrives.
controlled environment. The action of picking is performed by an induction manip-
Mahler et al. [9, 10] show the feasibility of directly predict- ulation robot consisting of a multiple-DoF arm with an end-
ing grasp qualities from point clouds with sufficient training of-arm tool (EoAT). The EoAT may consist of one or more
data. It is designed for a gripper and a single suction cup at suction cups. Depending on the EoAT design, each suction
the EoAT. Similar to the work by Araki et al. [2], the biggest cup may be controlled individually, only as groups, or only
drawback of this method is that the output is hard to interpret, all together.
and it is difficult to intelligently improve the performance Each pick is defined as a set of variables determining the
beyond increasing the training dataset size. However, many actions of the robot: a 3-D point in space (i.e., the desired pick
ideas from this work inspired our solution. point where the EoAT makes contact with an item’s surface),
Zeng et al. [18] focus on robotic manipulators with multiple the desired 3-D orientation of the EoAT at the pick point, and
EoATs. They propose having various sets of grasps for differ- a set of desired active suction cups on the EoAT. Note that a
ent EoAT types based on the robot’s perceived environment single package or even a package segment may be associated
and selecting the EoAT based on the highest predicted success with many candidate picks.
probability. It considers each affordance (e.g., which suction At each time t, we will use xt to represent the state of the
cups are active or inactive on the EoAT) as context rather than current scene and Pt to denote the complete set of possible
as a feature, producing many grasp sets to evaluate against the picks over the scene, determined by an elementary filtering
scene when there are many affordances. This drawback makes process (e.g., making sure the pick point is on an item).
the approach infeasible for real-time applications with EoATs Given a scene xt and any pick p ∈ Pt , we can construct
that have many affordances. a d-dimensional feature vector φ(xt , p) ∈ Rd that encodes
not only the parameters defining the pick p but also how the
Some scenes may be void of any picks estimated to succeed
pick relates to the scene. For example, it may include the
with high probability. In this case, Liu et al. [7] propose a
distance from the bottom of each suction cup to the surfaces
novel interactive exploration strategy that learns to push the
of the packages beneath, estimated from point-cloud data. See
objects around to obtain a better set of possible grasps in a
Section IV-B for the extracted features used in our current
complicated environment. While this method can potentially
deployment.
help in scenarios with no feasible picks or grasps, it is time- The role of the pick ranker is to use φ(xt , p) for each p ∈ Pt
consuming and cannot be directly deployed in fast-paced to define an ordering over the candidate picks Pt . This ordered
industrial tasks. list is passed through an final filtering step (e.g., checking the
Most of the research on grasp and pick quality and success feasibility of planning the arm motion without a collision),
prediction has been done in controlled lab settings, allowing and the robot executes the first feasible pick. The scene is
to hold assumptions such as having singulated, rigid, or 2- cleared if no viable picks are found (i.e., planning failure).
D shaped objects or only performing vertical picks. In order Once a feasible pick is executed, whether successful or not,
to have a working system in real-world uncontrolled settings, the process starts all over again on the next scene, which may
such as a package fulfillment center or a mail package sortation be a slightly modified or entirely new scene.
facility, the method should be able to overcome these limiting Ideally, the pick ranker would have perfect knowledge of
assumptions. the final filtering step and would dictate just a single pick to
In our proposed method, we have identified a selection of be executed. In practice, the final filtering process can vary by
relevant features and have developed models that can assess location and other constraints of the particular deployed robot,
the pick’s quality in a cluttered uncontrolled scene without which may not be known beforehand. To minimize complexity,
the limitations of other methods and within the industrial we only consider memoryless pick ranking systems: only the
computational and timing constraints. The methods are de- current scene is considered when choosing a pick, and there
ployed across a fleet of Robin manipulator robots in fulfillment is no effort to plan ahead a sequence of picks.
centers and have been responsible for picking over 200 million We assume there exists a function F : Rd → [0, 1] such that
packages during our evaluation period. To the best of our for a scene xt and any pick p ∈ Pt the probability that a pick
knowledge, our work is the only method for predicting the p will be successful is equal to F (φ(xt , p)) ∈ [0, 1]. Note that
pick quality and ranking of picks that can work with different this model assumes φ(xt , p) contains all necessary information
EoAT orientations, uncertain object material and properties, about whether a pick will be successful. This simplifying as-
and cluttered environments. sumption does not reflect that there may be unobserved factors
influencing pick success, such as the weight distribution inside information on the induct, such as the ground truth on the
the package. Extending our model to handle such partially success or failure, as well as information about the workcell
observed settings is ongoing work. (e.g., the station code, the type of manipulator arm and EoAT).
To maximize the probability of a successful pick at time t,
an ideal pick ranker would rank the picks of Pt in decreasing B. Feature Extraction
order of F (φ(xt , p)). Note that under this model, the success We compute a set of features for each induct using the meta-
probability F (φ(xt , p)) is agnostic to picks p that do not pass data, RGB, and depth images. Specifically, the camera data is
the final filtering process and thus can take an arbitrary value. processed by our perception system to generate segments of
In practice, we evaluate a surrogate for F (φ(xt , p)) (which the packages and tag each segment with an associated package
takes a non-negligible amount of computation), so ideally, Pt type label. Additional statistics are computed for each segment
would only include those picks that pass the final filtering using depth information (e.g., surface normals and the quality
process. of plane fitting). An overview of our perception system design
Of course, the true F is unknown, but we can estimate to extract the required features is shown in Figure 2.
it with data and an appropriate machine learning model (see
Section IV). Candidate Pick
(Pick Point, EoAT Orientation,
Point Cloud Cropping Score, Center, Normal Pick Metric and To Motion
Set of Active Cups)
and Plane Fitting Cup Distances to Plane, etc. Ranking Planner

IV. L EARNING TO E STIMATE P ICK S UCCESS

Calibrated Mask/Segmentation Score

This section describes our data-driven approach to estimat- Point Cloud Adjacency Graph Candidate Package Neighbors

Package Material
Computation Adjacency Graph, etc.
ing the probability of success for a given pick in a scene.

Segment
Masks
Segmentation
First, we provide the details for the training datasets used in
our work, and then we briefly outline the features extracted
for our model. Finally, we explain the details of the models RGB

developed for this project.

A. Training Dataset Perception System

We compiled three datasets from hundreds of actual induc-

tion cells in Amazon fulfillment centers. Due to the nominal Fig. 2: An overview of the perception system design to extract
success of pick ranking heuristic methods used in the past, an features for the learned pick success model.
independent and identically distributed (IID) random draw of
inducts would lead to a severely imbalanced dataset, with pick The segmentation module is a deep network based on Mask
success examples vastly outnumbering the failure examples. Scoring R-CNN [6] with a Swin-T backbone [8] for predicting
Therefore, we oversampled failures in all the training datasets the package material, instance segmentation masks, and the
to create a more balanced dataset. classification and segmentation scores. The rest of the features
• TrainDataset-Center: This dataset contains ~395K robotic
extracted for the model are directly computed from the input
inducts composed of 335,226 successful and 59,646 failed pick and point-cloud data.
examples. Additionally, due to the used heuristics, the Based on our feature importance studies, we identified the
location of the picks in the dataset is as close to the center following features as significant predictors:
of package segments as possible. • Package height: We believe this feature correlates with the
• TrainDataset-Random: For this dataset, ~41K randomly package’s momentum and, therefore, can impact the shear
selected inducts from TrainDataset-Center are replaced with force at the suction cups and the pick’s stability.
new inducts that were randomly distributed to be picked • Quality of plane fitting: We fit a plane on each segment,
anywhere on the packages’ segments with a higher chance and we speculate that a better plane fit correlates with a
of being close to center when a center pick is possible. The better seal between the suction cups and the package.
new set of inducts comprises 34,715 successful and 6,673 • Number of activated suction cups: More active suction
failed picks, and the total size of the newly-created dataset cups can mean a more stable pick, reducing the failure
is the same as TrainDataset-Center. probability.
• TrainDataset-Past: This dataset is compiled from historical • Alignment quality between the suction cups and the
data collected from inducts executed before the timeframe package surface: This feature is computed as the offsets
of TrainDataset-Center and TrainDataset-Random inducts. between the package surface normal vector and the normal
It contains ~230K inducts composed of 195,408 successful vector of the suction cups. We also expect this feature to be
and 34,482 failed examples. Similar to TrainDataset-Center, significant since a better alignment indicates a better seal
the location of the picks in the dataset is as close to the between the suction cups and the package surface.
center of package segments as possible. In addition to the above and other segment-specific features,
Each induction consists of the RGB image data captured we compute features that describe each segment’s relation-
by a camera at the top of the workcell looking straight ship with its surroundings, including the number of nearby
down, depth images, and metadata. The metadata includes segments and the adjacency graph features. To compute the
adjacency graph features, we construct a graph that captures can be used to generate ensembles. In anecdotal studies, we
the topological order of the package segments. This graph found that these ensembles capture uncertainty very well.
captures each detected segment’s relative height with respect Because our training data does not cover all possible picks
to its adjacent neighbor segments. Figure 3 shows an example for all scenes, capturing such uncertainty helps provide
where the numbers represent the relative position ranking of more conservative predictions.
the segment among its neighbors.
V. P ICK R ANKING
Let us assume that the robot has selected several picks in
the scene. For example, the system may generate one or more
picks per each object segment in the robot’s region of interest.
The model described in Section IV can output estimated
probabilities of successfully picking up and holding the desired
items for each candidate pick. Assuming the estimate closely
correlates with the actual probability of success, the picks with
a higher likelihood of success should be prioritized.
Ranking the picks based on their success probability esti-
mate provides two benefits: the throughput of the workcell is
improved due to the higher chance of picking each item up
on the first try, and, by picking up the “easier-to-pick” items
first, the “harder-to-pick” items become easier to pick (e.g.,
Fig. 3: Example of an adjacency graph for a cluster of items. the occluding or very close items are removed around them,
leaving them singulated).
C. Pick Success Model In our work, we rank the picks in two steps. First, the
The features described in Section IV-B, along with the picks are grouped by package segments, and the pick suc-
ground truth knowledge of induct success or failure, were cess probabilities or other heuristics are leveraged to rank
extracted for the inducts in the training datasets (Section IV-A). the segments. When picking success probability is employed
The AutoGluon library [4] was leveraged for training, model directly, we estimate the success probability of a segment as
i i
selection, and hyperparameter tuning for the pick success pre- Psegment = max Ppick , where Ppick is the success probabil-
i=1...n
diction binary classification task. Many models showed similar ity of the segment’s ith pick. Finally, once the segments are
performance, but a gradient boosting tree, specifically a Cat- ranked, for each segment, we order its picks based on their
Boost model [15], was among the top performers and was cho- success probability predictions. The two-step ranking is due
sen for our implementation. We also evaluated other machine to the logistics behind our system structure and the desired
learning libraries and modeling options, such as multilayer flexibility to try different methods for the two steps.
perceptron (MLP) with Scikit-learn [14]; however, models Figure 4 shows two examples of package rankings using
trained with AutoGluon showed superior performances. our model, where the flat and large package segments are
In contrast to our strategy of extracting interpretable (tab- prioritized over the crumpled and small ones.
ular) features, prior works have trained models that directly
predict the pick success from some combination of the input
RGB image, depth, and pick features. We also benchmarked
that approach but did not see a significant improvement over
our model’s performance. Moreover, our method has a few
advantages over the pixels-to-prediction approach:
1) Interpretability: We found it challenging to understand
what made a particular scene easy or difficult with the
pixels-to-prediction approach and why a specific pick
failed. In contrast, the tabular features we extracted were
easy to interpret and allowed us to characterize different
failure modes. This helps to identify weaknesses in the
training dataset to refine it later.
2) Computation: A gradient boosting tree is much faster to
train and evaluate than a typical image classification model.
This means that at deployment, we can evaluate many picks
very quickly. Fig. 4: Examples of ranking the packages based on the highest
3) Uncertainty quantification: The CatBoost package na- success probability estimate of their corresponding picks. A
tively supports sampling models from a posterior, which smaller rank number represents a higher priority.
 Figure 5 shows the predicted success probability of three B. Baselines and Experiments
picks on a deformable package for picking with the suction To evaluate the performance of different pick success mod-
cup arrangement illustrated in Figure 6(b). All three picks have eling options, we first consider the following two baselines:
two activated suction cups on the package. However, the model • AlwaysSuccess: Always predicting pick success;
appears to prioritize the pick configurations where the EoAT • BoostedTree-Past: Our pick success model described in
is less likely to collide with the surrounding packages and is Section IV-C trained with the historical TrainDataset-Past
more likely to succeed. data (see Section IV-A);
Historically, our robots were programmed to pick up pack-
ages at poses close to the package centers. However, there
are cases where the robots must choose picks further away
from the package center to avoid collisions, such as with other
packages or fixtures on the conveyor belt. To be able to see
the effect of choosing off-center picks, we trained two pick
success models:
• BoostedTree-Center: Our pick success model described in
(a) Ppick = 0.708 (b) Ppick = 0.756 (c) Ppick = 0.825 Section IV-C trained with the TrainDataset-Center data (see
Fig. 5: Estimated success of different picks on a deformable Section IV-A);
package. The circles correspond to the suction cups. The green • BoostedTree-Random: Our pick success model described
and red circles are the active and inactive suction cups. in Section IV-C trained with the TrainDataset-Random data
(see Section IV-A).
CNN-Center: Finally, we also report the performance
VI. E XPERIMENTS AND R ESULTS of an image-based model, which is a network trained on
The methods proposed in Sections IV and V have been TrainDataset-Center using RGB image crops around the target
deployed and tested in multiple Amazon sites worldwide. This packages (Figure 7).
section presents our testing conditions, experiment results, and
our analysis.
A. Hardware
The proposed method is implemented for Robin robot [1]
used in Amazon fulfillment centers. The main arm consists of
FANUC M-20iD/35 with six controlled axes, 35 kg payload, + +
28
56

0
1

10
and 1831 mm reach (Figure 6(a)). The EoAT consists of 8

20
1
112 112

2
fc3

51
224 1 fc2

2
51
suction cups arranged in an “X” configuration with a size of 28
fc1
56

56
128128 512
AvgPool
25 × 25 cm. Each suction cup can be controlled individually. ×4
Figure 6(b) illustrates this EoAT configuration.
2

2
11

11

11

64 64 64 256
Conv
4
22

3 ×3
Image Input

Fig. 7: Network architecture for extracting the RGB image

embeddings. The input is a 400 × 400 patch around the target
pick point (resized to 224 × 224), and the output is the pick
success prediction. The blocks before the AvgPool layer are
the first three convolution layers of a pre-trained Resnet50
model. The output of the second fully-connected layer (fc2)
is used as the embedding. The size of the fc2 layer output
(fc3 layer input) is set to 20, 48, or 96 for the desired image
embedding sizes.

The following testing datasets are designed to evaluate the

pick success estimation models:
(a) (b) • EvalDataset-Center: Consisting of ~60K picks that are
close to the package center. This dataset has an overall pick
Fig. 6: Robin robotic arm used in our experiments. (a) Ma- success rate of 94.40%.
• EvalDataset-Random: Consisting of ~38K picks randomly
nipulator arm. (b) A simulation of the end-of-arm tool design
with eight suction cups. chosen to be anywhere within the package segment. This
dataset has an overall pick success rate of 94.01%.
 For ranking segments on the actual robots, we considered Having experimented with these heuristics, we establish the
several heuristic approaches, including: following baseline and experiment to evaluate our proposed
pick ranking approach:
• Z-order: Ranking the picks by the package’s target surface
• Baseline: Topological order with Z-order as the tie-breaker
elevation. This heuristic is motivated by the assumption that
a package whose surface is at the top of the pile is easily for ranking segments; picks with EoAT poses with a higher
reachable, and the robot can avoid collisions or mistakenly number of active suction cups, and then the ones with the
pick other occluding packages when trying to pick it up. center of active cups closer to the center of the segment are
This heuristic is simple to implement but omits information given higher rank within segments;
• Experiment: Topological order with learned pick success
about actual occlusions and, in practice, fails for unreachable
picks and in instances where a portion of the package is estimation as the tie-breaker for ranking segments; picks
at the top of the pile, but the rest is buried under other within segments are also ranked with the learned pick
packages. In addition, since this method is only concerned success estimation.
with the elevation of the package’s surface, it is useless for To evaluate our approach without bias to a particular
packages lying on the conveyor belt. planning strategy, we allow the robots to select EoAT poses
• Package size: Ranking the picks by the package size (or randomly anywhere as long as the poses are within 30 cm
segmentation area). This heuristic assumes that picking distance from the center of the packages on the estimated
packages with a larger visible area will have higher success package surface plane.
and that moving out these larger packages first will help Moreover, additional large-scale experiments were per-
declutter the scene, making picking smaller packages easier. formed through A/B tests on the Robin fleet to assess the
A major issue with this heuristic is when smaller packages performance of different methods. Based on the possible
lie on or overlap with larger ones. This can result in heuristic approaches, we established the following baselines
collisions with other packages, difficulty lifting a larger to evaluate against our proposed pick ranking approach:
package from under smaller packages, or picking more than 1) TopoZ-Center: Topological order with Z-order as the
one package simultaneously, resulting in holding or multi- tie-breaker for ranking segments; picks within segments
pick failures. chosen close to the center of the package’s segment with
• Topological order: Ranking the picks based on the order higher rank given to the EoAT poses with a higher number
of occlusion of the packages. Picks to package faces that of active suction cups;
appear unoccluded get the best score. Picks on package 2) Z-Center: Same as TopoZ-Center, but only Z-order used
surfaces that are only occluded by unoccluded packages get directly for ranking segments;
the next highest score, and so on. This heuristic is mod- 3) TopoZ-Random: Same as TopoZ-Center, but picks within
erately simple to implement and prioritizes the unoccluded segments are chosen randomly anywhere in the package’s
packages to improve the success probability. However, it segment, with a higher chance for picks closer to the
fails to recognize the unreachable package surfaces and the segment’s center.
occluded packages with unoccluded surfaces, and similar to For evaluation of the ranking methods through learned pick
the Z-order method, it does not differentiate between the success estimation, we designed the following experiments:
picks on the same package surface. Moreover, this method 1) TopoLPR-Center: Topological order with learned pick
heavily relies on an accurate perception system to compute success estimation as the tie-breaker for ranking segments;
the segment overlaps. picks within segments chosen close to the center of pack-
age’s segment with learned pick success estimation used
Any combination of the above heuristics may also be em-
for ranking;
ployed. For example, a topological order method can be used
2) LPR-Center: Same as TopoLPR-Center, but learned pick
as the primary ranking criteria, with the Z-order method as
success estimation is directly used for ranking the seg-
the tie-breaker when two packages have the same topological
ments;
ranks. Some other heuristic approaches are also worth men-
3) LPR-Random: Same as LPR-Center, but picks within seg-
tioning, such as the measure of how quadrilateral a package
ments can be chosen randomly anywhere in the package’s
is or the confidence score given by the instance segmenta-
segment.
tion method. These approaches try to indirectly measure a
package’s occlusion or deformation level but have their own C. Evaluation Metrics
drawbacks and challenges. Due to the rarity of failed inducts, our evaluation datasets
The heuristics above only rank the segments to be picked are severely imbalanced. As a result, evaluating the classifica-
and cannot differentiate between various picks inside a seg- tion accuracy does not serve as an informative metric for this
ment. Once the segments are ranked, a reasonable heuristic task (for example, a baseline that always predicts pick success
approach for ranking picks inside a segment can be giving a will get over 94% accuracy).
higher score to the picks closer to the center of the segment On the other hand, the main objective of a pick success
and the ones with a higher number of activated EoAT suction classifier is to use its output estimates for ranking the picks so
cups. that picks with a higher chance of success are ranked higher.
 1 1
We choose Receiver Operating Characteristic (ROC) Area
Under the Curve (AUC) score as the metric for evaluating the 0.8 0.8

True Positive Rate

True Positive Rate

pick success models. Mathematically, the ROC-AUC score is 0.6 0.6

the same as the probability of a classifier ranking a randomly-

CNN-Center CNN-Center
0.4 0.4
BoostedTree-Past BoostedTree-Past
chosen positive example higher than a randomly chosen neg-
BoostedTree-Center BoostedTree-Center
ative example, i.e., P (score(x+ ) > score(x− )) (see [5] for 0.2
BoostedTree-Random
0.2
BoostedTree-Random
AlwaysSuccess AlwaysSuccess
the proof). Therefore, when all the successful picks in the 0 0

testing datasets are ranked higher than all the failed picks, the 0 0.2 0.4 0.6
False Positive Rate
0.8 1 0 0.2 0.4 0.6
False Positive Rate
0.8 1

ROC-AUC score would be 1.0, and when all the failed picks
are ranked higher than successful picks, the ROC-AUC score (a) (b)
would be 0. Therefore, the ROC-AUC score is a good metric Fig. 8: ROC curves of all models described in I tested on eval-
for evaluating the pick-ranking ability of different models. uation datasets (a) EvalDataset-Center and (b) EvalDataset-
To evaluate the pick ranking system on the actual hardware, Random.
we also compute the percentage of picks when the robots fail
to transfer a package from the conveyor belts to the mobile
robots. We call this metric as failure rate. always close to the center when evaluated on EvalDataset-
Center. On the other hand, there is a more significant margin
D. Results between BoostedTree-Random and BoostedTree-Center when
All models introduced in Section VI-B were evaluated on they are evaluated on EvalDataset-Random. This shows that
both testing datasets EvalDataset-Center and EvalDataset- our pick success model can interpolate between centered picks
Random. Table I presents the ROC-AUC scores of these and picks that are more off-center, and including the off-
models with confidence intervals. center picks in the training dataset helps with predicting pick
success for larger varieties of picks, which can be beneficial for
TABLE I: ROC-AUC scores with confidence intervals of packages that are hard to reach and the robot has to select from
different models for the pick success estimation task. a list of off-center picks. Finally, the RGB image-based model
CNN-Center performs much worse than the other modeling
Model EvalDataset-Center EvalDataset-Random
options. Given CNN-Center makes predictions only based on
AlwaysSuccess 0.5 (0.5, 0.5) 0.5 (0.5, 0.5) the package appearance, it shows that additional information
BoostedTree-Past 0.725 (0.717, 0.732) 0.807 (0.799, 0.815)
about the picks is critical even if the robots attempt to pick
BoostedTree-Center 0.755 (0.748, 0.761) 0.802 (0.792, 0.810) up the packages close to the center.
BoostedTree-Random 0.758 (0.752, 0.765) 0.848 (0.840, 0.855)
CNN-Center 0.570 (0.560, 0.579) 0.703 (0.693, 0.712) The Experiment ranking method (see Section VI-B) was de-
ployed for production on Amazon’s fulfillment center robotic
fleet and has been used for picking up over 200 million inducts
As seen from Table I, all the machine learning models with a success rate of 98%. We analyzed ~180K random
beat the naive baseline AlwaysSuccess that always predicts robotic inducts performed using Baseline and Experiment
pick success. Additionally, all models perform better on ranking methods to validate our proposed method. Table II
EvalDataset-Random compared to EvalDataset-Center. From summarizes the results, which shows that Experiment method
Figure 8, we observe that the ROC curves on the EvalDataset- improves the pick success rate by about 1.18%. This 23.7%
Center dataset (Figure 8(a)) are flatter than the ROC curves reduction in failures, when deployed at a large scale (e.g.,
on the EvalDataset-Random dataset (Figure 8(b)) for middle millions of picks per day performed on our fleet), has a
range false positive rates. This indicates that there are more significant impact on the operation costs.
pick failure examples that are hard to differentiate from the
pick success examples in the EvalDataset-Center dataset. TABLE II: Test results for validation of pick-ranking learned
Upon further investigation of the datasets, we found that a method.
sizeable portion of pick failure examples is due to factors not
included in our feature set, such as suction cup degradations. Method Total Picks Pick Success Pick Failure Success Rate
We believe the proportion of such examples is more significant Baseline 89,162 84,718 4,444 95.02%
when the robot action space is more constrained, such as Experiment 90,127 87,700 3,427 96.20%
always attempting to pick packages close to the center.
When comparing models trained with data from different To better understand the cases where these methods rank
time ranges (i.e., BoostedTree-Past vs. BoostedTree-Center the segments differently, we present a qualitative comparison
and BoostedTree-Random), we observe the model performance for two cases that assist with understanding the behavior of
improves when it is trained with more recent data. the learned pick success method. In general, given similar con-
Additionally, we find that the model trained with picks ditions for two packages, the learned pick success estimation
sampled anywhere within the segment (i.e., BoostedTree- method seems to prefer flatter surfaces and packages with less
Random) slightly outperforms the model trained with picks occlusion while disliking the packages close to the conveyor
wall or at hard-to-reach angles.
1) Case 1: Figure 9 shows an induct with its segment
ranking results for three methods: topological order with Z-
order for tie-breaking, topological order with learned pick
success estimation method for tie-breaking, and the learned
pick success estimation method. It can be seen that the learned
approach prioritizes the packages closer to the center of the
conveyor belt (away from the conveyor walls), where the robot
is less likely to have difficulties with reaching the pick at the (a) (b)
desired angle or colliding with the conveyor wall.

(c) (d)
(a) (b) Fig. 10: Qualitative comparison of different segment ranking
methods. (a) Another example induction scene with a marked
ROI (the blue rectangle). Results for: (b) topological order
of segments with Z-order used for tie-breaking, (c) topolog-
ical order of segments with learned pick success estimation
used for tie-breaking, and (d) learned pick success estimation
method directly used for segment ranking.

TABLE III: Results of A/B experiments for pick-ranking

(c) (d) methods.
Fig. 9: Qualitative comparison of different segment ranking Method Total Picks Failed Picks Success Rate
methods. (a) An example induct scene with a marked ROI (the TopoZ-Center 1,158,353 89,378 92.28%
blue rectangle). Results for: (b) topological order of segments Z-Center 1,157,739 89,866 92.24%
with Z-order used for tie-breaking, (c) topological order of TopoZ-Random 1,158,479 109,193 90.57%
segments with learned pick success estimation used for tie- TopoLPR-Center 1,156,697 83,535 92.78%
breaking, and (d) learned pick success estimation method LPR-Center 1,160,005 72,789 93.73%
LPR-Random 1,157,342 79,820 93.10%
directly used for segment ranking.

2) Case 2: Figure 10 shows an induct with its segment

ranking results for the same methods as Case 1. Comparing and Z-Center). Interestingly, the best improvement comes from
the ranking for segments ranked 2 and 3 in Figures 10(b) the more aggressive approach where we directly apply the
and 10(c), it is evident that the learned method prioritizes learned pick success estimation for ranking (LPR-Center).
the packages with less occlusion where the collision with the Given that the heuristic ranking methods heavily depend on
occluding packages is less likely. On the other hand, when the the heights of the packages, this suggests that promoting high
constraints of topological order are removed, it can be seen packages can lead to more challenging picks, such as tall
in Figure 10(d) that the learned method slightly prefers the but unstable packages. On the other hand, the pick success
packages with flatter surfaces (the box over the deformable model considers the package height only as an input feature
packages) where the chances of holding failure are lower. along with other information such as the package position,
Finally, to find the best method, the large-scale A/B test surface normal, and adjacency graph features. Therefore, the
was deployed across the fleet with a small percentage of total pick success model can reason better and deprioritize unstable
inducts allocated to each experiment group in Section VI-B. packages. It is also worth noting that if the picks are chosen
Table III summarizes the results of the A/B experiments. randomly, the pick success improvement from the heuristic
The results show that using the learned pick success esti- ranking method (TopoZ-Random) to the learned pick success
mation to rank the segments (i.e., TopoLPR-Center and LPR- estimation ranking method (LPR-Random) is even more sig-
Center) improves the pick success for the robots compared nificant (i.e., from 90.57% to 93.10%).
with the manual heuristic ranking methods (i.e., TopoZ-Center The pick success estimation model deployed in these A/B
experiments predicts the pick success probability by taking manufacturing using pick-and-place systems – a review.
the average of the predictions from five CatBoost models. The Robotics and Computer-Integrated Manufacturing, 51:
number of trees in the five CatBoost models is 2236, 1069, 222–229, 2018. ISSN 0736-5845. doi: 10.1016/j.
799, 1464, and 1208 respectively. For all five models, the depth rcim.2017.12.003. URL [Link]
of the trees is 6, and we used a learning rate of 0.05. science/article/pii/S0736584517301758.
[4] Rasool Fakoor, Jonas W Mueller, Nick Erickson, Pratik
VII. C ONCLUSION Chaudhari, and Alexander J Smola. Fast, Accu-
In this paper, we presented a large-scale deployed system for rate, and Simple Models for Tabular Data via Aug-
package manipulation, which estimates the pick success using mented Distillation. In Advances in Neural Informa-
a machine learning model. We demonstrated the effectiveness tion Processing Systems, volume 33, pages 8671–8681,
of this system by evaluating it on over 200 million picks and 2020. URL [Link]
comparing it to heuristic baselines. file/[Link].
We believe that the recent developments in vision trans- [5] J A Hanley and B J McNeil. The meaning and use of
formers combined with a large amount of induction data from the area under a receiver operating characteristic (ROC)
our robotic fleet can improve our image-based network and curve. Radiology, 143(1):29–36, 1982. doi: 10.1148/
may provide more valuable image embeddings, enhancing the radiology.143.1.7063747. URL [Link]
prediction quality of the overall method. radiology.143.1.7063747.
Additionally, going through the mistakes made by our [6] Zhaojin Huang, Lichao Huang, Yongchao Gong, Chang
model, we realized that a sizable portion of them are due to Huang, and Xinggang Wang. Mask Scoring R-CNN. In
hardware issues such as a dysfunctioning suction cup. In the 2019 IEEE/CVF Conference on Computer Vision and
future, we intend to leverage the developed pick success model Pattern Recognition (CVPR), pages 6402–6411, 2019.
for monitoring the health and analyzing the errors in our robot doi: 10.1109/CVPR.2019.00657. URL [Link]
fleet. [Link]/document/8953609.
[7] Huaping Liu, Yuhong Deng, Di Guo, Bin Fang, Fuchun
ACKNOWLEDGMENTS Sun, and Wuqiang Yang. An Interactive Perception
The work would not have been possible without the support Method for Warehouse Automation in Smart Cities.
of the wider Amazon Robotics team. More specifically, the IEEE Transactions on Industrial Informatics, 17(2):830–
authors would like to thank previous and current members of 838, 2021. doi: 10.1109/TII.2020.2969680. URL https:
the Robin and Janus teams, who established the underlying //[Link]/document/8970574.
technology, provided support, and helped shape and deploy [8] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei,
these ideas. The authors would like to give special thanks Zheng Zhang, Stephen Lin, and Baining Guo. Swin
to David Oreper and Sicong Zhao for enabling deployment Transformer: Hierarchical Vision Transformer using
and to Shuai Han, Qiujie Cui, Mansour Ahmed, and Andrew Shifted Windows. In 2021 IEEE/CVF International
Marchese, whose help was critical in the realization of this Conference on Computer Vision (ICCV), pages 9992–
work. 10002, 2021. doi: 10.1109/ICCV48922.2021.00986.
URL [Link]
D EMONSTRATION AT RSS 2023 [9] Jeffrey Mahler, Florian T Pokorny, Brian Hou, Melrose
During the presentation of this paper, we intend to have a Roderick, Michael Laskey, Mathieu Aubry, Kai Kohlhoff,
live demonstration of our evaluation or production workcells. Torsten Kröger, James Kuffner, and Ken Goldberg. Dex-
The demonstration will present live webcams from different Net 1.0: A cloud-based network of 3d objects for ro-
sites across the globe showing Robin robots picking and bust grasp planning using a multi-armed bandit model
placing incoming packages. with correlated rewards. In IEEE International Confer-
ence on Robotics and Automation (ICRA), pages 1957–
R EFERENCES 1964, 2016. URL [Link]
[1] Amazon Science. Amazon Robotics: Robin, 2022. URL 7487342.
[Link] [10] Jeffrey Mahler, Matthew Matl, Vishal Satish, Michael
with-a-world-where-things-are-changing-all-around-it. Danielczuk, Bill DeRose, Stephen McKinley, and Ken
[2] Ryosuke Araki, Tsubasa Hirakawa, Takayoshi Ya- Goldberg. Learning ambidextrous robot grasping poli-
mashita, and Hironobu Fujiyoshi. MT-DSSD: Multi- cies. Science Robotics, 4(26):eaau4984, 2019. doi:
task deconvolutional single shot detector for object de- 10.1126/scirobotics.aau4984. URL [Link]
tection, segmentation, and grasping detection. Advanced org/doi/abs/10.1126/scirobotics.aau4984.
Robotics, 36(8):373–387, 2022. doi: 10.1080/01691864. [11] A. Morales, E. Chinellato, A.H. Fagg, and A.P. del
2022.2043183. URL [Link] Pobil. Experimental prediction of the performance of
full/10.1080/01691864.2022.2043183. grasp tasks from visual features. In Proceedings 2003
[3] Andreas Björnsson, Marie Jonsson, and Kerstin Jo- IEEE/RSJ International Conference on Intelligent Robots
hansen. Automated material handling in composite and Systems (IROS 2003) (Cat. No.03CH37453), vol-
 ume 4, pages 3423–3428 vol.3, 2003. doi: 10.1109/ 978-3-030-91135-5. doi: 10.1007/978-3-030-91135-5 5.
IROS.2003.1249685. URL [Link] URL [Link] 5.
document/1249685. [17] Nazib Sobhan and Abu Salman Shaikat. Implementation
[12] Douglas Morrison, Peter Corke, and Jürgen Leitner. of Pick & Place Robotic Arm for Warehouse Products
Learning robust, real-time, reactive robotic grasping. The Management. In 2021 IEEE 7th International Conference
International Journal of Robotics Research, 39(2-3):183– on Smart Instrumentation, Measurement and Applica-
201, 2020. doi: 10.1177/0278364919859066. URL https: tions (ICSIMA), pages 156–161, 2021. doi: 10.1109/
//[Link]/doi/10.1177/0278364919859066. ICSIMA50015.2021.9526304. URL [Link]
[13] Huong Giang Nguyen, Marlene Kuhn, and Jörg [Link]/abstract/document/9526304.
Franke. Manufacturing automation for automotive [18] Andy Zeng, Shuran Song, Kuan-Ting Yu, Elliott Don-
wiring harnesses. In 8th CIRP Conference of Assem- lon, Francois R. Hogan, Maria Bauza, Daolin Ma,
bly Technology and Systems, volume 97, pages 379– Orion Taylor, Melody Liu, Eudald Romo, Nima Fazeli,
384, 2021. doi: [Link] Ferran Alet, Nikhil Chavan Dafle, Rachel Holladay,
254. URL [Link] Isabella Morona, Prem Qu Nair, Druck Green, Ian
pii/S2212827120314761. Taylor, Weber Liu, Thomas Funkhouser, and Alberto
[14] Pedregosa, Fabian, Varoquaux, Gaël, Gramfort, Alexan- Rodriguez. Robotic pick-and-place of novel objects
dre, Michel, Vincent, Thirion, Bertrand, Grisel, Olivier, in clutter with multi-affordance grasping and cross-
Blondel, Mathieu, Prettenhofer, Peter, Weiss, Ron, domain image matching. The International Journal of
Dubourg, Vincent, Vanderplas, Jake, Passos, Alexandre, Robotics Research, 41(7):690–705, 2022. doi: 10.1177/
Cournapeau, David, Brucher, Matthieu, Perrot, Matthieu, 0278364919868017. URL [Link]
Duchesnay, and Édouard. Scikit-learn: Machine Learning doi/10.1177/0278364919868017.
in Python. Journal of Machine Learning Research, 12: [19] Kun Zhang and Hua Zhang. Design and implementa-
2825–2830, 2011. doi: 10.5555/1953048.2078195. URL tion of automatic apple crating robot technology. In
[Link] 2021 IEEE 2nd International Conference on Big Data,
[15] Liudmila Prokhorenkova, Gleb Gusev, Aleksandr Artificial Intelligence and Internet of Things Engineer-
Vorobev, Anna Veronika Dorogush, and Andrey Gulin. ing (ICBAIE), pages 617–621, 2021. doi: 10.1109/
CatBoost: unbiased boosting with categorical features. ICBAIE52039.2021.9389974. URL [Link]
In Advances in Neural Information Processing Systems, [Link]/abstract/document/9389974.
volume 31, pages 1–11. Curran Associates, Inc., 2018. [20] Zhao Zhang, Anand Kumar Pothula, and Renfu Lu. A
URL [Link] Review of Bin Filling Technologies for Apple Harvest
[Link]. and Postharvest Handling. Applied Engineering in Agri-
[16] Mir Salahuddin and Young-A Lee. Automation with culture, 34(4):687–703, 2018. ISSN 0883-8542. URL
Robotics in Garment Manufacturing, pages 75–94. [Link]
Springer International Publishing, Cham, 2022. ISBN