Built Robotics executives are fond of saying that their autonomous system for construction equipment, like dozers and excavators, might be further along than many autonomous vehicles. In fact, CEO Noah Ready-Campbell insists you’ll see autonomous vehicles in controlled industrial environments — like construction sites — before you see level 5 driverless cars on public roads. That may be in part because autonomous construction equipment often operates on privately owned land, while public roads face increased regulatory scrutiny.

“There’s a quote that ‘Cold fusion is 20 years in the future and always will be,’” Ready-Campbell told VentureBeat on Wednesday. “I think there’s a chance that that might be true for level 5 self-driving cars as well.”

That might’ve seemed like an absurd thing to say back when autonomous driving first entered the collective imagination and companies established their intention to solve AI’s grand autonomous vehicle challenge. But Waymo now takes billions from outside investors, and the delay of major initiatives like GM’s Cruise and taxi service and Ford’s autonomous driving program call into question the progress automakers have made on autonomous vehicles.

One thing Ready-Campbell credits autonomous vehicle companies with is generating excitement around AI for use in environments beyond public roads, like on construction sites.

“We were the beneficiaries of that when we did our series B last year,” he said. “I definitely think construction benefited from that.”

From computer vision systems and drones to robots walking and roving through construction projects, Built Robotics and a smattering of other companies are working in unstructured industrial environments like mining, agriculture, and construction to make autonomous systems that can build, manage, and predict outcomes.

To take a closer look at innovation in the field, the challenges ahead, and what it’s going to take to create fully autonomous construction projects in the future, VentureBeat spoke with startups that are already automating parts of their construction work.

Autonomous excavators and heavy machinery

Built Robotics creates control systems for existing construction equipment and is heavily focused on digging, moving, and placing dirt. The company doesn’t make its own heavy construction equipment; its solution is instead a box of tech mounted inside heavy equipment made by companies like Caterpillar, Komatsu, and Hyundai.

Built Robotics VP of strategy Gaurav Kikani told VentureBeat that the company started with autonomous skid steers — the little dozers that scoop up and transport sand or gravel on construction sites. Today, Built Robotics has autonomous systems for bulldozers and 40-ton excavators.

“We have a software platform that actuates the equipment that takes all the data … being read by the sensors on the machine every second and then makes decisions and actuates the equipment accordingly,” Kikani said.

Built Robotics focuses on earthmoving projects at remote job sites in California, Montana, Colorado, and Missouri — far removed from human construction workers. Autonomous heavy equipment monitored by a human overseer tills the earth in preparation for later stages of construction, when human crews arrive to do things like build homes or begin wind or solar energy projects. In the future, the startup, which raised $ 33 million last fall, wants to help with more infrastructure projects.

Kikani and Built Robotics CEO Ready-Campbell say the company is currently focused on projects where there’s a lot of dirt to move but not a lot of qualified operators of heavy machinery.

Calling to mind John Henry versus the machine, Kikani said human operators can go faster than a Built-controlled excavator, for example, but machine automation is meant to provide consistency and maintain a reliable pace to ensure projects finish on schedule.

Built Robotics combines lidar with cameras for perception and to recognize humans or potential obstacles. Geofencing keeps machinery from straying outside the footprint of a construction site. Excavators and dozers can work together, with dozers pushing material away or creating space for the excavator to be more productive.

“The fleet coordination element here is going to be critical. In Built [Robotic]’s early days, we really focused on standalone activities, where you have one piece of equipment just on its own taking care of the scope. But realistically, to get into the heart of construction, I think we’re going to start to coordinate with other types of equipment,” Kikani said. “So you might have excavators loading trucks [and] autonomous haulage routes where you have fleets of trucks that are all kind of tracking along the same route talking to each other, alerting each other to what they see along the route if conditions are changing.”

“I think the trickiest thing about construction is how dynamic the environment is, building technology that is pliable or versatile enough to account for those changing conditions and being able to update in real time to plan to accommodate for that. I think that is really going to be the key here,” he said.

Equipment operated by systems from companies like Built Robotics will also need computer vision to recognize utility lines, human remains, or anomalies like archeological or historically important artifacts. It’s not an everyday occurrence, but construction activity in any locale can unearth artifacts that lead to work stoppage.

Drones, robots, and computer vision

Drones that can deploy automatically from a box are being developed for a variety of applications, from fire safety to security to power line inspection. Drones hovering above a construction site can track project progress and could eventually play a role in orchestrating the movement of people, robotic equipment, and heavy machinery.

In a nod to natural systems, San Francisco-based Sunflower Labs calls its drones “bees,” its motion and vibration sensors “sunflowers,” and the box its drones emerge from a “hive.”

Sensors around a protected property detect motion or vibrations and trigger the drones to leave their base station and record photos and video. Computer vision systems working with sensors on the ground guide the drone to look for Intruders or investigate other activity. Autonomous flight systems are fixed with sensors on all four sides to influence where the drone flies.

Sunflower Labs CEO Alex Pachikov said his company’s initial focus is on the sale of drones-in-a-box for automated security at expensive private homes. The company is also seeing a growing interest from farmers of high-value crops, like marijuana.

Multiple Sunflower Labs drones can also coordinate to provide security for a collection of vacation homes, acting as a kind of automated neighborhood watch that responds to disturbances during the months of the year when the homes attract few visitors.

Stanley Black and Decker, one of the largest security equipment providers in the United States, became a strategic investor in Sunflower Labs in 2017 and then started exploring how drones can support construction project security and computer vision services. Pachikov said Sunflower’s security is not intended to replace all other forms of security, but to add another layer.

The company’s system of bees, hives, and sunflowers is an easy fit for construction sites, where theft and trespassing at odd hours can be an issue, but the tools can do a lot more than safeguard vacant sites.

When a Sunflower Labs drone buzzes above a construction site, it can deploy computer vision-enabled analytics tools for volumetric measurement to convert an image of a pile of gravel into a prediction of total on-site material.

Then tools from computer vision startups like Pics 4D, Stockpile Reports, and Drone Deploy can provide object detection, 3D renderings of properties for tracking construction progress, and other image analysis tools.

Companies like Delair take a combination of data from IoT sensors, drone footage, and stationary cameras from a construction project to create a 3D rendering that Delair calls a digital twin. The rendering is then used to track progress and identify anomalies like cracks or structural issues.

Major construction companies around the world are increasingly turning to technology to reduce construction project delays and accident costs. The 2019 KPMG global construction survey found that within the next five years, 60% of executives at major construction companies plan to use real-time models to predict risks and returns.

Indus.ai is one of a handful of companies making computer vision systems for tracking progress on construction sites.

“We can observe and use a segmentation algorithm to basically know every pixel — what material it is — and therefore we know the pace of your concrete work, your rebar work, your form work and [can] start predicting what’s happening,” Indus.ai CEO Matt Man told VentureBeat in a phone interview.

He envisions robotic arms being used on construction sites to accomplish a range of tasks, like creating materials or assembling prefabricated parts. Digitization of data with sensors in construction environments will enable various machine learning applications, including robotics and the management of environments with a mix of working humans and machines. 

For large projects, cameras can track the flow of trucks entering a site, the number of floors completed, and the overall pace of progress. Computer vision could also follow daily work product and help supervisors determine whether the work of individuals and teams follows procedure or best trade practices.

“Imagine a particular robotic arm can start putting drywall up, then start putting tiles up, all with one single robotic arm. And that’s where I see the future of robotics […] To be able to consolidate various trades together to simplify the process,” Man said. “There could be armies of robot-building things, but then there is an intelligent worker or supervisor who can manage five or 10 robotic arms at the same time.”

Man thinks software for directing on-site activity will become more critical as contractors embrace robotics, and he sees a huge opportunity for computer vision to advance productivity and safety in industrial spaces.

Stanford University engineers have explored the use of drones for construction site management, but such systems do not appear to be widely available today or capable of coordinating human and robotic activity.

“Having all these kinds of logistical things run together really well, it’s something I think AI can do. But it’s definitely going to take some time for the whole orchestration to be done well, for the right materials to get to the right place at the right time for the robot to pick it up and then to do the work or react if some of the material gets damaged,” Man said. “In the current construction methodology, it’s all about managing surprises, and there are millions of them happening over the course of the whole construction plan, so being able to effectively manage those exceptions is going to be a challenge.”

One construction robotics platform to rule them all

Boston Dynamics, known for years as the maker of cutting-edge robots, also entered construction sites last year as part of its transition from an R&D outfit to a commercial company.

Like Sunflower Labs’ drones, Boston Dynamics’ four-legged Spot Mini with a robotic grasping arm acts as a sensor platform for 360-video surveys of construction projects. Capable of climbing stairs, opening doors, and regaining its balance, the robot can also be equipped with other sensors to track progress and perform services that rely on computer vision.

An event held by TechCrunch at the University of California, Berkeley last month was one of the first opportunities Bay Area roboticists have had to convene since the pandemic precipitated an impending recession. Investors focused on robotics for industrial or agricultural settings urged startups to raise money now if they could, to be careful about costs, and to continue progress toward demonstrating product-market fit.

Speaking on a panel that included Built Robotics CEO Ready-Campbell, startups debated whether there will be a dominant platform for construction robotics. Contrary to others on the panel, Boston Dynamics construction technologist Brian Ringley said he believes platforms will emerge to coordinate multiple machines on construction sites.

“I think long-term there will be enough people in the markets that there will be more competition, but ultimately it’s the same way we use lots of different people and lots of machines on sites now to do these things. I do believe there will be multiple morphologies on construction sites and it will be necessary to work together,” Ringley said.

Tessa Lau is cofounder and CEO of Dusty Robotics, a company that makes an automated building layout called FieldPrinter. She said there’s a huge opportunity for automation and human labor augmentation in an industry that currently has very little automation. Systems may emerge that are capable of doing the work of multiple trades or on-site activity management, but Lau said there can be nearly 80 different building trades involved in a construction site. Another problem: Construction sites are by definition in various stages of fairly constant change. The dynamic nature of construction sites — where there is no set or static state like you might find in a factory — presents another challenge.

“I think the flip side is if you look at a typical construction site, it’s chaos, and anyone with a robotics background who knows anything about robotics knows it’s really hard to make robots work in that kind of unstructured environment,” she said.

Forget the word “robot”

One thing the TechCrunch panelists agreed on is that robots on construction sites won’t succeed unless the people working alongside them want them to. To help ensure that happens, Lau suggested startups slap googly eyes on their robots because people want to see things that are cute or beloved succeed.

“Our customers are rightfully concerned that robots are going to take their jobs, and so we have to be careful about whether we are building a robot or … building a tool,” Lau said. “And, in fact, we call our product a FieldPrinter. It’s an appliance like a printer. It uses a lot of robotic technology — it uses sensors and path planning and AI and all the stuff that powers robotics today, but the branding and marketing is really around the functionality. Nobody wants to buy a robot; they want to solve a problem.”

Built Robotics CEO Ready-Campbell wholeheartedly agreed, arguing that even a thermostat can be considered a robot if the only requirement to meet that definition is that it’s a machine capable of manipulating its environment.

Last month, just before economic activity began to slow and shelter-in-place orders took effect, the International Union of Operating Engineers, which has over 400,000 members, established a multi-year training partnership with Built Robotics. Executives from Built Robotics say its systems operate primarily in rural areas that experience skilled labor shortages, but Ready-Campbell thinks it’s still a good idea to drop the term “robot” because it scares people. Opposition to construction robotics could also become an issue in areas that see high levels of unemployment.

“That’s how we position Built [Robotics] in the industry, because when people think of robots, it kind of triggers a bunch of scary thoughts. Some people think about The Terminator, some people think about losing jobs,” he said. “It’s an industry that really depends on using advanced machinery and advanced technology, and so we think that automation is just the next step in the automation of that industry.”