Tuesday 11 July 2023

CASCADE BRIDGE Case Study

As part of the EPSRC CASCADE Programme on aerial robotics, we studied semi-autonomous control for drones inspecting structures.

What kinds of structures?

As the name suggests, we set out to target bridges, and we were especially fortunate to try some early flight trials at the Clifton Suspension Bridge, by kind permission of the Clifton Suspension Bridge Trust.  We were able to prove our ability to fly in the bridge environment (during a planned closure) and to detect the bridge using a simple, lightweight LIDAR sensor.
Unfortunately, rather a lot happened between 2018 and 2023, which meant we were unable to return to the bridge for a full trial of our final system.  Later trials took place at the Snowdonia Aerospace Centre in Llanbedr, North Wales.

What is semi-autonomous control?

For us, it means keeping the pilot in control of the system, while automating appropriate tasks such as monitoring the vehicle health and checking distances.  The drone remains within visual line of sight of the pilot, and the pilot remains vigilant for air risks (other aircraft) and ground risks (e.g. people or vehicles nearby).
This partnership demands a simple interface between the pilot and the control system.  It must not become more complicated for the pilot to figure out what the control system is doing.  For this reason, we adopted simple "traffic light" feedback from the control to the pilot, and a simple "yes/no" input from the pilot to the control.  The concept is inspired in part by railway signalling systems.
Under the hood, we use a behaviour tree to create a modular controller, combining many different checks and actions in a flexible, re-usable way.

What research did we actually do?

  • Developed and validated the concept
  • Designed a set of behaviour tree templates to build different missions
  • Showed how formal safety arguments could be constructed for this combination of human and automation
  • Deployed the system in both immersive simulation and real flight
  • Conducted a human factors trial of the system (results pending)

Where can I learn more?

  • The original case study design is here.  (We didn't do all of it...)
  • A paper on the semi-autonomous drone control concept
  • A paper on mission templates in behaviour trees
  • A paper on safety analysis
  • A paper on the immersive simulation deployment, in collaboration with CASCADE partners at the University of Manchester
  • A good introduction to behaviour trees can be found in the book by Colledanchise and Ă–gren.

Can I try it myself?

Our Python library for control of drones is available on GitHub.

Acknowledgements

Research at the University of Bristol FlightLab by PhD student Hirad Goudarzi supported by technical specialist Duncan Hine.  Supervised by Arthur Richards and Tom Richardson.  Supported by the UK Engineering and Physical Sciences Research Council as part of the CASCADE programme grant EP/R009953/1



Wednesday 19 August 2015

Should Driverless Cars have Steering Wheels?

Note: these are purely the personal views of Arthur Richards, and do not necessarily represent the views of the VENTURER project or any of its members.

I'm delighted to be part of the VENTURER consortium, trialling autonomous cars in Bristol and South Gloucester.  My role in VENTURER at Bristol Robotics Lab is to deliver a decision-making system for the car, figuring out where to move next given sensor information from other partners' equipment.  We'll not be developing a complete car automation solution, but we will give it just enough brains to do something interesting in the test scenarios.  In particular, we'll be looking at motion planning as a tool to resolve those scenarios that don't really appear in the Highway Code, but occur often enough round Bristol.  Consider a bus trying to pass a recycling lorry opposite some roadworks and with a car trying to pull out of a drive in between...

Since VENTURER started I've had a few regular questions come up.  Google's recent accident raises plenty more (and the reports give us a fascinating view into how the car sees the world),  Here is my FAQ.

Will self-driving cars have steering wheels?

Apparently lots of people want them, because they don't trust technology and want to remain in control.   That seems fair enough: given the frustrating experience of automated checkouts, we're accustomed to the idea that automation needs a bit of supervision.  In a way it makes life easier for us technology developers too: if the driver is still in charge of the vehicle, alert and ready to take control at a moment's notice, then the technology is really only there to assist the driver and doesn't have to be capable of handling all situations.  In short, it doesn't have to be as reliable, because there's a human backstop if it fails.  We call this capability "handover" - the ability to hand over control from computer to human, or vice versa.

But in my view, handover is a fallacy for two reasons.  First, if I still have to remain as alert and responsible for the vehicle as if I'm driving it, what's the point in all this shiny expensive kit to automate it?  Second, if I'm responsible for stepping in when it does something wrong, I need to be very sure that I can recognize "wrong" and know better what "right" is.  That means I have to understand what the automated driving system is doing and why, so I can tell when it's gone wrong.  So, now, not only do I still need to be as alert and capable a driver as I ever was, but I've got a whole load of extra techie stuff to learn about how this fancy gadget works.

You can see more problems with handover if you think about the liability issues.  So let's suppose I'm in my car in auto-drive mode.  I'm as alert as I am now, but I can still easily miss something: 42% of all accidents identify "driver failed to look properly" as a contributing factor. My car picks up on a movement of another car that I've not seen, and reacts.  I don't like the reaction, take control, and have an accident.  Who's at fault?

Meanwhile, if my car is actually pretty good in auto-drive mode, I get used to letting it run.  My own driving gets worse as I'm out of practice, and since I so rarely need to do anything in the car, like it or not, my attention wanders.  So, I'm not as alert as I need to be, and I'm not as practised at what to do.

But can't the car tell me when it needs me to take control?

This too is a fallacy.  The handover capability is there so that if something goes wrong with the auto-driving system, you can take over control of the car.  If the auto-drive is capable of detecting when it's gone wrong, then it should also be able to fix the problem.  This is like saying "I'll call you if I have a problem with my phone."

So there won't be a steering wheel?

Hold on!  I've argued that taking control during driving is a bad idea.  However, a self-driving car is going to be a complicated bit of kit, with lots of sensors, computers and electronics all over it.  Things will break.  Given everything I've said above, the car needs to be able to get you safely to the side of the road, without your intervention, in the event of a failure, and this kind of "fail-safe" engineering is common.  However, a secondary manual driving capability from kerbside to garage is probably essential.  It would be awful if someone only had to back gently into your bumper and that knocked out a couple of sensors and immobilized the whole car.  Your self-driving car is going to be covered with gadgets, and they make most sense to be at the periphery where they've got the best view, so they're going to get bumped in traffic.  The steering wheel here is in a role similar to that of a get-you-home spare tyre: it's not for regular use, but it's there if there's no alternative.

So there is a steering wheel, but I shouldn't be tempted to use it?

Pretty much.  The wheel is there if you want to use it or if, due to some failure, the car can't be started in auto-drive mode.  It's not there for you to take over from auto-drive mode while in motion.

But I still want to be in control?  How can I gain trust in the auto-drive before I'm ready to go all the way?

This is the best question yet, and one to which I don't have a good answer.  This kind of trust only comes from accumulated experience.  I'm reasonably happy to sit on a flight without having watched the pilot fly a few times before, or studied the design of the aircraft.  Instead, I base my trust in the history and culture of the airline and aviation business.  So, somehow, all the stakeholders in auto-driving cars need to find a way to earn this trust.  Easy to say...

Will my self-driving car be more efficient?  Will it help the environment?

Interesting question.  There are many different ways in which this technology will impact the environment, especially vehicle emissions.  I've not run the numbers but we can think about the qualitatively at least.  For the purposes of this thought experiment, assume that a self-driving car means one that can drive itself with no human driver attention, but is otherwise identical to any other car.  (People tend to think of self-driving cars as futuristic electric vehicles, but you could apply that technology to manually driven cars just as well.  The same goes for regenerative braking or auto-engine-stopping in traffic.  Let's keep those separate from self-driving technology.)

On the plus side, how will self-driving cars help the environment?

The automated driving system ought to improve efficiency by optimizing acceleration and braking profiles and gear changes if necessary.  Not sure how significant these are though.  Optimizing speed would be a more meaningful impact, but how often is speed free to be optimized, rather than set by congestion? Not that much.

One significant positive impact could be in reduction of accidents.  If self-driving really does lead to fewer accidents - and if it doesn't, there was very little point - then there could be significant reductions in knock-on congestion.  Also, if the reduction in accidents could reduce the carbon costs of building spares and new ones - because there are fewer cars damaged or written off in accidents.

Long term, after a significant uptake of self-driving cars, congestion could be ameliorated by cooperative routing strategies.  Self-driving technology goes hand-in-hand with the connected car concept, and although you could add vehicle-to-vehicle (V2V) or vehicle-to-infrastructure (V2I) capability to any car, they seem pretty likely to come together.  Imagine an optimized response to high-bandwidth traffic reporting.

OK, the negative side: how will they hurt the environment?

Potentially, self-driving cars could lead to an increase in traffic levels, with all the knock-on congestion and environmental detriment that would bring.  We all make value-driven choices when choosing modes of transport.  I often prefer to take a train instead of drive because I can work on the way, or arrive refreshed instead of tired after driving.  The train's rarely cheaper though, and I still have to cover the door-to-station bits myself.  But, if my car can drive itself, I can go door-to-door and still get the work done or rest.  Trouble parking at the far end?  No problem - I'll send the car off to park itself and call it back later on.

Also, full self-driving opens up the possibility of personal car transport for people who can't actually drive manually.  Would we still expect people to stop using their own cars due to eyesight, illness or just plain old age?  Invariably there will be limits, but if self-driving technology emerges, it would be very harsh to withhold it from those who could benefit most in terms of independence.  There are some interesting possibilities for the school run too, but don't let's get too carried away.

The main point is that fully-automated driving technology removes a number of disincentives to the use of your own car for any given journey: the need to be capable (and perhaps qualified) to drive the car; the mental exertion of driving the journey; your own time spent driving the journey; and potentially some of the time and effort in parking at the other end.

I've not run the numbers on any of these, and it'd be a big project to do properly anyway.  What would be the net effect?  Hard to say, but it's not "given" than self-driving cars will be good for the environment.

If two people run into the road from different sides, and the car can only physically miss one of them, how will it choose?

There are lots of variations on this question, involving various combinations of unfortunate potential victims.  It turns out that this is a variant of the well-studied trolley problem in ethics, and some argue that this question is at the heart of self-driving car technology.  I disagree.

That's not to say I'm flippant about safety.  Far from it.  I will do all that I can to make sure that any decision-making system I develop has minimal chance to ever end up in a situation where it might have to choose who to kill.  Then I'll do some more to reduce that chance.  But once in that situation, things have already gone badly wrong.  The boundary of what's acceptable vs. what isn't lies before that judgement point is reached.  Because things always go wrong sometimes, I cannot guarantee that we'll never cross that boundary, but as an engineer my priority is to minimize the chance of crossing it.

There's nothing in the highway code about how a human driver should choose, or any other legislation that I'm aware of.  It's suggested that human drivers can be forgiven for poor choices in the trolley problem because they don't have the time to consider it ahead of time - but then if it's so important, why isn't it in the driving test?

Personally I'm also a little chilled by the idea of programming my computer to make value judgements about human lives.  Presumably if it got it right, it'd tell me my journey isn't worth the risk anyway.  I'd rather spend my time improving the car's ability to anticipate where people might run out in front of it.  I hope I've not been at all flippant about this question, but it really is a distraction.

Thursday 16 July 2015

Tutorial Presentation on MPC for Aircraft

Thank you to Pantelis Sopasakis from IMT Lucca for inviting me to participate in a tutorial session on Model Predictive Control for Aerospace at the European Control Conference in Linz, Austria.  Presentation topics ranged from spacecraft rendezvous to UAV guidance.  I limited my talk to fixed-wing aircraft, but covered three different case studies:

  • Nonlinear MPC for a UAV avoiding oncoming aircraft
  • Mixed-integer linear MPC for an aircraft flying in a forest
  • Mxed-integer linear MPC for air traffic flow
Click here or on the image below to get the slides from my talk.



And to close, a nice picture of the Danube by night!

Untitled

Monday 12 January 2015

Flying a Drone in the Royal Institution Christmas Lectures

In December 2014, the first cohort of FARSCOPE Centre for Doctoral Training students developed an automated drone cymbal player to take part in an RI Christmas Lecture.  FARSCOPE Director Arthur Richards reflects on the experience.

The Royal Institution approached Bristol Robotics Lab in November 2014 to ask if we could contribute a drone to their robot orchestra project, inspired by this YouTube video and others.  The brief was quite broad: get a drone to play something somehow.  We agreed to investigate and see what we could do.  In particular, we made this a "crash project" for the new FARSCOPE students, perhaps a poor choice of term under the circumstance, but highlighting the emphasis on simplicity and timeliness of whatever we came up with.

Drones are tough to control with great accuracy, so we quickly narrowed our ideas down to percussion.  My personal, and slightly ironic, wish was to have a drone play the triangle This urge was quickly subdued, both by the drowning of the triangle sound by the motor noise, and the alarming response of the drone when affixed with a beater and flown into a triangle.  It was clear that some sort of transmission would be required to strike the instrument.  Electrical transmission was ruled out, as we wanted it to be obvious that it was really the drone making the noise.  Therefore, we had a requirement for a mechanism that would convert some motion of the drone into motion of the beater striking the instrument.

CDT Student Dave wheels the drone kit into the Faraday Lecture Theatre
Around this point, I started to have unsettling thoughts about a drone flying into a live audience of youngsters, never mind in front of TV cameras.  Consultation with the production team indicated that a perspex shield would give reflection problems, and our ideas for cages made fishing wire just seemed pretty outlandish.  So, the drone would have to be tethered.  This neatly solved the mechanism debates as well, as the tether would serve to actuate a beater.  So, with a few weeks to go, we had a concept: a drone tethered to a pivoting arm, which would crash into a cymbal when pulled upwards.  All this could be mounted on a trolley, conveniently also serving as transport and landing pad.

Meanwhile, other details had been falling into place.  Students had been building QAV250 quadrotor drones and equipping them with Naze32 flight stabilizers.  These turned out to be excellent machines, mechanically robust, well-stabilized, and easy to fly for beginners, so very suited to our needs.  Some Arduinos were programmed to take commands from our ROS environment and convert them to PWM streams for the trainer port of our radio transmitters, with an added emergency stop functionality.  With our Vicon motion capture system tracking the drone, we were able to close the loop.  One day we may share the radio control bridge and PID tuning packages with the ROS community, but it's the nature of crash projects that things aren't done as tidily as they could be.

Students put finishing touches to the drone landing pad in the RI library
With a van full of spare drones, computers, markers, cameras, tools and fishing wire, we headed to London for the first of two long days rehearsing and filming.  With plenty of practice in the lab and enough spare kit to do the whole project twice, we would be confident, but nevertheless the first successful cymbal clap when hooked up to the orchestra was a great relief all round.  As so often in robotics, the thing that came closest to stopping us was a flaky network.

One surprise was yet to come: the producers liked the drone so much they wanted to make more of it, so we quickly contrived a demo for it to follow an external target.  For comedy reasons and as a nod to the computer vision world, it had to be a teapot.  It's a great credit to the students involved that an extra function was written into the software at about an hour's notice, but that went on to work beautifully.

FARSCOPE students at the control station, backstage at the RI.
You can see the finished product late on in the video at the RI Channel: Sparks Will Fly: How to Hack Your Home  For the cynics, there was absolutely no cheating involved.  There was a manual transmitter out back but it was never used during filming, and the only manual intervention was me pressing the button to land the drone, giving lecturer Danielle a bit of a fright.  (Ground effect when the downwash hits a surface makes it hard to fly a drone very low and keep it tightly controlled - you're better off just cutting the power once low and slow.)  Everything you see in the video is genuinely automated, with students pressing only a start button and switching the teapot tracking on and off.

So what do we learn from all this?  Drones will clearly not be great musicians in the near future.  Still, the students learnt a wide range of practical skills through this exercise, from ROS use to drone construction.  Time management, teamwork and the importance of testing were also reinforced, although this was far from a good model of systems engineering.  Finally, it was great for those of us running FARSCOPE in its first year to explore the flexibility and responsiveness that the Centre for Doctoral Training gives us.

Thursday 20 February 2014

Show and tell

I've been doing lots of demonstrations and presentations lately - here's a collection.

First was a very interesting workshop at the ASAP group in Nottingham, supported by the LANCS initiative, talking about applications of operations research to air transport problems.  Agenda and talks are available for download here, including my report of our work on aircraft taxi optimization and air traffic flow management.

Next was a more high level event at Southampton, the Autonomous and Advanced Systems Showcase event.   This covered various opportunities and initiatives at the intersection of aerospace and autonomous systems, ranging from market opportunities to the latest research details.  Extensive video reporting can be found here, including my report on the work of Bristol Robotics Lab in this area.

The Duke of York visited BRL on Feb 10th and was treated to, among other things, a display of our autonomous flying robots in a building exploration scenario.  The photos are here.

And finally, I was asked to comment on the recent Amazon UAV delivery proposals by a packaging trade magazine.  The article is here.

Monday 27 January 2014

FARSCOPE Centre for Doctoral Training - NEW WEBSITE

The new and settled home for information on the FARSCOPE Centre for Doctoral Training in Robotics and Autonomous Systems is: farscope.bris.ac.uk

Thursday 19 December 2013

FARSCOPE

In 2014, the Bristol Robotics Lab (BRL) will be launching a new Centre for Doctoral Training in robotics and autonomous systems.  The Centre will offer a joint PhD degree run by BRL's two partner universities, the University of Bristol and the University of the West of England.  The Centre is funded by EPSRC following their CDT 2013 competition.
The new CDT is entitled "Future Autonomous Robotic Systems Centre of PhD Education" - FARSCOPE.  In time FARSCOPE will have its own website with full information on the programme and how to apply.  Until then, the latest information and news for prospective students will be found here.

Theme

FARSCOPE is all about making robots more adaptable.  Instead of doing the same jobs over and over in  large factories, they'll help us in our homes, work alongside us in small businesses, and deal with hazardous situations while we keep a safe distance.  To make all this happen, we face three challenges:
  •  robots being part of society, interacting with people, machines or other robots, and handling all their unpredictable behaviour
  •  robots operating in uncertain environments, sensing, mapping and moving where existing maps and plans are inaccurate or unusable.
  •  requiring different roles of robots with regular changes, for small batch jobs in small businesses where two days downtime to reprogram isn't an option.
Together, these challenges make up our theme of adaptability.  The technologies involved are diverse and the applications are many, but anything to do with the three challenges above is within the remit of FARSCOPE.

Programme

FARSCOPE enhances the "traditional" PhD content of individual research with taught content on a range of robotics topics and more general skills.  Not only does this better prepare you for PhD research, but it gives you a broad view of robotics and autonomous systems that we believe is essential. There's so much to be learnt by seeing what's been done in different fields and applications: manufacturing benefits from autonomous exploring ideas and spacecraft can exploit UAV breakthroughs, to name just too.  In short, we want you to be adaptable too, in your skills and what you can do with them.

Year 1

  • Research methods training
  • Seminars in modern robotics methods
  • Robotics, mechanics and programming
  • Robotics context and applications (industry delivered)
  • Robot intelligence and systems
  • Specialist robotics topics (chosen from list of options)
  • Group robot project (eg IMAV contest, robot soccer or Mars rover field test)
  • Initial research project
  • Communications training and research presentation

Year 2

  • PhD research
  • Industry study workshop
  • Innovation and entrepreneurship
  • Complementary skills training

Year 3

  • PhD research
  • Industry study workshop
  • Partner placement (optional: opportunities at partner universities in Europe, Asia, North America or partner companies in the UK and Japan)
  • Public engagement training and group activity

Year 4

  • PhD research
  • Complementary skills training (including thesis preparation)
  • FARSCOPE conference presentation

More Information

More details will appear here and on our website as it becomes available.  For questions or if you're interested in applying, please contact arthur.richards@bristol.ac.uk