Embedded Security Research Panel @ Intent Summit

so welcome to the embedded security panel for in 10th 2022.

so before we get started let’s speak a bit about what we’re doing here first of all what is an embedded system so I’m

going to quote from Wikipedia it’s a bit long so we just took the first paragraph an embedded system is a computer system

a combination of a computer processor computer memory and input output peripheral devices that has a dedicated

function within a larger mechanical or electronic system basically for our purposes it’s anything

which isn’t a client or a server running Linux iOS or Mac or Android okay but

even those contain embedded systems in them today but why is this interesting for a

security conference so who is targeting embedded devices and why nation states and offensive security

vendors are now targeting different types of embedded systems you might have

heard on attacks on networking telephony and cellular equipment the Ukrainian grid attacks had some embedded attacks

in them and multiple vendors now offer Espionage tools for embedded systems

cyber criminals are also interested in embedded systems you might have heard of Ransom Wars in hospitals and information

theft from different Medical Systems and also vehicle theft most vehicle theft today is done by attacking the embedded

systems in the vehicle and of course the Maori and Jailbreaker communities are still interested in

consoles in vehicles chip tuning and infotainment hacks and in Mobile security and of course security

researchers which is US everyone here and this is happening now and everywhere and you hear about in the news and it

will just get worse as time goes by so our agenda for today is some questions for our panelists and then

we’ll have some questions from the audience finally we will summarize our discussion and all that needs to be in

45 minutes actually 38 minutes and panelists please remember to explain

every non-trivial term you use the first time you use it and with that let’s introduce the panel

yesen [Applause]

and I’m Benny misels [Applause] okay great check your microphones

check test test perfect so shakhar please introduce yourself uh hi everybody thanks for coming to our

panel uh I’m sure I’m senior director of security research at jfrog uh I manage

the teams that work on CV analysis malware analysis my previous company we

were doing automated analysis of embedded firmware and that’s where I’m coming from

go hi everyone I’m gal I’m a security researcher and I’m doing mostly

vulnerability research on embedded systems in the past I used to work on automotive and I did a vulnerability

research for electronic control units and now I’m doing other types of

embedded Securities including Oasis hypervisors basements and so on

and so that’s it about me hello my name is Scott Lawson and I work

at huge amsterdad at the secure mobile networking lab and there I mostly look into sorry I mostly look into mobile

devices and wireless chips and I do a lot of reviews engineering because most

of this is not documented and find vulnerabilities and write about them

okay so I’m Betty maisels I’m lead solution architect at cymotive cyber security company for automotive for the

last four and a half years I’ve been doing penetration testing on different embedded systems part of the time of gal and recently I transferred over to be a

security architect because I’m fed up of finding the same vulnerabilities again and again and I want to fix them in the source

with that let’s go to our first question so our first question for all of our

panel members is what device do you like hacking what are this typical security issues for this class of device let’s

start with Fiesta so as I already said I look a lot into mobile devices like your iPhone your

Android phone and usually you think like okay there is like it’s one operating

system if iOS is secure everything is secure but actually in that smartphone

there are tons of chips and those are these embedded system that we are now talking about and if they are insecure

then also your whole system becomes insecure like even without any escalation into an operating system if

your Bluetooth chip is insecure uh the attacker might be able to listen to your

phone calls or extract contacts and the same goes for a base band or for a Nitro vipenship so whatever embedded systems

you have in your phone if they are insecure everything becomes insecure

come so one of the things that I like to hack in the past was an arm hypervisor so we

usually hear about like hypervisors with Intel virtualization but in embedded systems most of the systems will have

arm architecture instead of intellore some other stuff that you might have heard before about hypervisors and so

the way I see it is in the end the vulnerabilities that you would look for will be very similar to different types

of hypervisors but the different technology aspect of the virtualization

was really interesting in my perspective so I was able to look at a few different

type of vitals and this is something that is available and integrated in

embedded systems nowadays um yeah so for my side uh I I guess uh

anything that uses u-boot uh uh for those that know you boot is uh open

source bootloader and a ton of embedded devices use it uh but it’s very interesting because uh

it really comes with a lot of unsafe defaults and then every vendor tries to like write custom code to make it safe

which is really hard so every time it’s like a CTF basically to see what custom

code they added that makes you able to hack it so really any type of device that uses

your boot you probably can always get a shell but it’s also always an interesting challenge I would say

I think the cool thing is that now we can find systems which actually have all three of these in one system and that

makes it totally awesome okay so our next question for yaskan gal can

we even build secure and better devices Kyle please start so in general like we most of us know

um every device have vulnerabilities in the end so what I would suggest

um is that because we know that in some point in the future vulnerabilities will exist on

every device so the most important part is to be able to update the devices

remotely so one of the things that I would suggest if someone is interested

in creating secure embedded devices is to be able to have an over-the-air

update and to be able to patch vulnerabilities and bugs when you have them instead of having a static device

that is not updated for years and years and one of the things that is also

important is that because embedded devices are integrated in air production lines and Healthcare staff so in the end

it’s important to have something that will not shut down everything so the

updates are something that I think will be very crucial if you want to have a secure embedded system

yeah I think updates are really crucial um and yeah everyone wants to hear like

this device is secure but it doesn’t exist but what you can see for embedded devices especially is that mitigations

that we have seen in other parts they are really having a hard time to make it

into embedded so for example secondaries they really had a long time until I saw

the first time with that Canary is enabled in a Bluetooth firmware and the

same goes for like aslr that’s usually not enabled on embedded devices and also

rust is now slowly making its way for embedded devices but it just takes time

it’s slower so probably in a few years we can say okay all the mitigations that

we know already now have made it also to embed it but let’s give it a try yeah we’ve ever seen

the future so maybe a follow-up to that question and what about Hardware reduction

Hardware defenses for let’s say electromechanical systems

yeah so uh from what I have seen like the typical like mobile device wireless

chips they are not considered something that has to be secured against fault

injection so maybe if you have like a crypto wallet or something that’s worth it but if they want to bypass a secure

bootloader in some of the common Wireless chips that’s pretty much possible and also the question is like

what is your adversary model like if someone already has your phone opened uh

do they attack the wireless chips or do they attack on an iPhone like the secure Enclave processor or some secure element

or some trust Zone on arm or something cool

anyway okay so our next question for yesterday so what is broken in embedded cyber

security and how can we fix it uh yeah so it’s a really good follow-up

and I’m gonna touch about some stuff that Jessica also said uh I think there’s a few categories here uh first

of all in embedded it’s still like in a different League than mobile and servers like it’s still

much less secure in my opinion first of all it’s the extensive use of

low-level languages I believe like a lot of times it’s written in C C plus plus you know when they’re not using grass

they’re not using python or JavaScript Etc uh and then you know you’re just susceptible to buffer overflows which is

think that you’re not even supposed to be susceptible anymore or command injections because there’s no secure API

things like that so first of all like the low level languages second uh like

use cassette mitigations like almost all the bsps today like they give

you an SDK can you explain what a bsp is yeah sorry the board support package it’s the uh both like physical and uh

software uh that you get when you’re working on a specific embedded device or a specific board from the vendor from

the OEM so um you get like a GCC uh version which

doesn’t even enable stack canneries and SLR and everything that this can set by default uh and then Ubuntu you know you compile

with GCC and Clank you get it right off the bat so it’s absurd like it’s mitigations from 20 years ago and you

still don’t get them by default and I will say very shortly the third thing maybe is uh a lot of times in embedded

people are not using open source as much as they need to they’re just trying to re-implement everything and then you get

bugs so these are the top three for me I guess

yeah but I mean so with open source you really had an interesting point because often in embedded devices so I I know

there is a GPL and you shouldn’t copy any Linux source code into your I don’t

know Wireless chip yeah but uh I’ve actually seen cases where you have a

vulnerability in Linux and I don’t think that the developer of the embedded chip copied the whole source code but they

probably took a look either into the source code or the specification is really bad and they just make the same

mistakes also in embedded but the tools for testing embedded devices and then

also finding and verifying vulnerabilities they are really bad and also this whole process of getting a

patch out like for an operating system that has OTA updates asgar said like three months

to push out a security patch it’s really doable but for embedded devices uh often

even the manufacturer they don’t know to which customers they sold which chips

with which firmware and rolling that out oh wow so uh I think what we need to fix

first is really updates so that’s getting back to what God Said and then

yeah the points that you mentioned so do you have any examples of vendors

which are doing this correctly are already on track to implementing these mitigations or updates

I mean I wouldn’t say that there is like correct or wrong

um so one thing that I’ve seen for Wi-Fi and Bluetooth updates is that like

bigger customers like Samsung or Apple who have like hundreds of thousands of

devices just of only one specific chip they will get updates rather fast so for

them this update process works but if you go to iot devices

that will take much longer so I think it’s kind of fixed and done well for the

big customers and then also their customers who are end users like all of you but

for the iot devices I don’t know and will regulation help here there’s new regulations in Europe for example

for updates of iot devices or default passwords and stuff like that

it might have but it’s by the chain like

uh and also it would make the products way more expensive if you really have someone who is constantly checking like

is there now an update for this chip and everything and then you I think you have to request patches especially if you’re

the changes to the firmware you have to request Patches from the tier one going up all the way and then saying hey I

only have 100 iot devices sold to customers can you provide a special patch for me that will take ages

thank you okay our next topic is education and

Industry uh these questions are for yes first of all for yes what is the

difference between security research in Academia and Industry so in Industry I think I would correct

me if I’m wrong yeah but in Industry I think uh the common thing is to do pen testing so you have a customer and the

customer wants a pen test and they have a checklist and then you said like I tested all the interfaces of this

component and everything that we Mark critical and so on so you have like a full checklist and in Academia you know

like I I take a walk I get a shower and then I come up with this one weird idea

and I say this is a new attack technique and I’m no going to try this and I’m

going to pawn at least like three or four systems with that and going to show like this is really a new thing and I

don’t care about like the overall product security I want I will never provide the company with a pen testing

list or something I would just say like this is a novel approach and also I

don’t care about payment or back Bounty I only care about publishing this

I think that another thing that we can add to that is that indeed there is a

lot of as that kind of research that you described in the industry but in the end

there are some companies who want to also publish some research for Innovation which is more similar to the

Academia but it really depends who in the end is the customer like if the

customer is the company so then you publish stuff in order to increase The Prestige of the company to get new

customers but there is also customers that pays for a security testing that in

the end requires some methodology of going through some of the things and

interfaces in order to make sure they are closed and secured so it really depends but I think that in the Academia

it’s more like individual person perspective and that’s like what is the

whole um customer or company needs out of this research so okay

our next question is forgot how can I get into embedded security research

this is a question I get a lot and I would say that the way that I

started is that I started to learn on my own and the thing that I would I would

suggest to anyone who wants to start embedded research or even any other type of research is to look at other people’s

white papers and talks in the end we try

to approach a new research objective but sometimes when you’re new to the subject

you are not always sure where to start like what is the right methodology to do those things this is something you are

not you cannot learn it by reading code you cannot learn it by looking at

specific components this is something you need to see and other people’s work to understand so in the end what I would

suggest is read a lot of white papers and white papers that has not only just

like here is a bug but mostly white paper us that have like the pauses of

the research how to approach the problem and another thing that I would suggest

is that learning alone is very hard because in the beginning you have a lot

of motivation you want to learn everything you want to know everything and a lot of the people through the time

will start to have less motivation so I would suggest find a friend and choose

something you want to learn together and then start with it that way you can kind of give different perspective you can

give motivation to each other and even if you have stuff that you don’t understand you have someone to talk with

that’s used in the same condition as you are and this is very um positive for the learning process and

that would be my two suggestions in general this guy you want to add something

yeah maybe to get started um because security research is is really special and I have the feeling

like I I do a lot of teaching and an I.T security Master offered by our University and I have the feeling many

people just go there and say I want to hack something I want to do security but

before you can do security research you have to understand a system in depth so

become a specialist in whatever like embedded security so

like get some embedded devices set up to us so first understand it and then try

to get into security like try to build a program load it in githra and reverse

engineering try to build a program that has security issues try to exploit them

I would agree totally I also think that it’s much more accessible than it used to be both platforms for researching

embedded but also platforms for developing for embedded used to be you had vendor tools which would run only on

Windows and they would be awful and extremely expensive these days much of embedded development is done on free

tooling if you want to ask me more about that maybe come after and we can tell you more about that but to our next

topic so now we’re going to move to some questions which are addressed individually to our panelists each one

in their expertise so first will be Shahar what is the role of Automation in embedded security research

sure um so I’m going to split it into three categories I guess

um first of all is uh because you can attack an embedded device in a lot of ways so first of all

I would say like supply chain security the second I would say is configuration issues and the third I would say uh zero

days a good old you know vulnerabilities in the first party uh and I’ll try to make it a bit quickly

uh so basically for supply chain it’s kind of similar to just scanning non-embedded images I would say like you

want something automatic that will just you know tell you all the CVS

um there’s a lot of free tools today so even you know you don’t even have to buy something as an entry level so it’s

great uh even uh for example uh for example for golang uh which some more advanced I

guess the embedded devices started using uh they even publish in their vulnerability database uh the exact

functions that are vulnerable and not just you know oh if you have this Library You’re vulnerable so it’s really

cool you can they have an automated tool that you know only if you’re using the specific function which is vulnerable it

will tell you uh That You’re vulnerable and it’s free so for supply chain you

know it can be totally automated and I know not all embedded companies do it the second is configuration issues the

issue with embedded I believe is you know companies are not using Helm charts and things like you know in the web

world they’re just compiling nginx not even using a package or something like that dropping it into there and just

going from default config to changing some stuff and then you you know it runs this route you’re left with default

password so all of these things can also be just scanned automatically there’s

also free tools to do that and there’s no reason not to use automation for all of these configuration issues

uh and I think the third thing like I said zero days um because of the use of low-level

languages and the lack of mitigations there’s still a lot of issues like uh like low level low hanging fruits sorry

uh like command injection things that are just you know basically you you you

give meta characters in the URL and you have a command ejection or stuff like that and you know we have soft engines

like source code and assets engines which are great at finding these things if you scan like Windows kernel or

Android or something like that um you’re gonna be flooded with false positives probably with these tools but

when you actually scan in a better system you have a much higher chance to find these low hanging fruit because

again low level languages everything is re-implemented so I believe like this is

still extremely viable like uh it’s like finding zero days with that so I’m specifically interested in the

second topic you mentioned the misconfigurations yeah off the top of your head can you mention some tools that our audience could use for that

um I’ll have to think for some we usually use the some paid ones so I have to think of uh some free ones I’ll try

to think uh uh there’s uh for example check SEC like c-h-e-c-k SCC uh that’s

really good for uh like a very basic check that you’ve enabled all mitigations like compiler mitigation so

what Jessica said SLR their stack canneries uh fortify Source all of that

it’s like you know you can just check for free very easily uh with the

specific configurations of specific tools it becomes harder with three uh

tools but I’m sure like I’m just not really updated with that like I’m sure you know there’s some good free ones but

uh like check stack is a very good one that I can recommend for like the mitigations yeah

thank you so next is for Jessica on the topic of Publications and bounties

how should we publish vulnerabilities in embedded systems

[Music] so there is also some back body programs that ask you to not publish and I think

publication is more important than bug Bounty so try to publish probably it

takes a longer time for embedded because the patch process is taking so long so it might take you like six months

instead of three months until you’re able to but yeah blog posts are fine if it’s

something that is systematically in many devices then you can of course also go for conference talk or or papers or

something but yeah first of all publish and also try to overcome something by a

vendor so there’s definitely vendors in embedded because patching takes so long

they just try to shut up like the researchers to silence them thank you

oh sorry oh yeah so why should vendors also be interested in the vulnerability

disclosure process so what I have seen is when I was reporting some stuff in in Bluetooth

firm various that not always everything that I reported to the tier one is then

also being propagated to their customers like I find a vulnerability I write up

some details and then this write-up is probably shortened or I don’t know what exactly happens in between like I’m not

in between but uh when I then also inform major customers who are affected

by the same vulnerability I saw that this gets faster and like things are

being fixed so probably especially to to reach the small customers who don’t have so many devices so I think it’s

especially for the smaller customers not the big vendors I mean they know about the vines but everyone down they they

should be interested in this stuff being published because it’s probably the only way for them to get to know and to fix

their systems so our next question is about embedded Technologies for gal

embedded systems have so many different Technologies how can I as a security researcher or analyst deal with all of

them so this is a big problem that we all encountered at some point and I think

that Although our instinct is like yeah we want we want to research code we want

to use either let’s reverse everything so because they there are so many different Technologies the first person

the first thing you should do before you dive into code is kind of understand where you are standing and which means

like reading documentation reading and other types of information you have on

your device and you would be surprised that in some cases looking online on

stuff would save you a long time of reversing a component and sometimes there are some there is documentation on

the protocol on some other firmware and that is similar to what you are looking

at and in the end a lot of the things are based on something that Engineers

need to do first so in these cases there will be also documentation

and another thing that I would say is that because we want to in case we want

to do vulnerability research you shouldn’t just like start you need to map a deck vectors first so this is

something that kind of helps you with the different technologies that you have is to focus on what’s really important

instead of like going everywhere look at everything because you need to focus on one thing and because the different is

so big so you need to decide which path is the most profitable to your research

and so I would suggest a tile look for the stuff before you dive into the code

try to understand if you have somewhere or some resource you can read the

information you need for the research project and also you can find sometimes

the symbols and stuff that can help you even though you have different technology sometimes firmware have

symbols online so this way is also a saving time when you deal with different

types of Technologies okay we will now take questions from the

audience please queue up in front of the microphone over there I believe and please add us a question to the

specific panel member you’re interested in or otherwise if it’s for the whole panel for the whole panel please ask only a single question to allow as many

people as possible to participate and if you have multiple questions please return to the end of the line after each

question okay so please anyone who wants to please come up to the microphone

foreign

don’t be shy

Aires do you have a question

I guess I’ll start thank you for the very interesting insights so um there was some talk about computer

law earlier in the panel and at least in Israel right now

some kind of pen testing is legal and if you’re attacking a computer server which

somebody else has do you think the laws about embedded devices

are friendly to hackers and should they be changed it’s the question for everybody

would like to take that please please go all right so

I think in better devices it depends on what specifically like if you can get

the embedded device let’s say a dev kit or a phone that has embedded ships I

think it’s actually quite friendly for research because like when you talked about servers and when you send a

request to a server there is internet in between and logging in between but when

you hack an embedded device you will not Target anything that’s live on the internet and at least like I mean I come

from Germany so we have different laws but in Germany when you target a life

system and you disrupt the services knowingly or even do stuff like just

iterating some ideas or something that’s already an attempt that could end you in

paying a lot of money or even ending up in prison depending on on what happens so hacking servers

really difficult but embedded devices it’s local it’s kind of

okay so what so something that can go wrong is of course like when you want to report and publish but the overall

research process is not illegal in that sense I think that I can add that

[Music] um with service it’s indeed really different in Israel today is regulations

regarding that kind of research but if you talk about embedded device and then you have a local setup so this is

something else and the problem that I see was researching and better devices is that a lot of the companies doesn’t

have bug bounties so it reduced the amount of researchers that want to do

this kind of research and to try and find vulnerabilities and publish them in the end it brings a lot of researchers

and it brings a lot of attention to devices and when you have an available

bug Bounty program so this is something I think the vendors should try to adapt

to the new age of kind of research we have today and so this is from my point

of view to add to what jiska said thank you please hello uh first off

amazing panel absolutely loving the conference today uh quick question uh

over the course of your embedded security analysis of devices whether it

be Bend walking or whatnot what is the most common vulnerability that you come across and uh second question what’s

your favorite vulnerability that you come across uh yeah so I guess this would be good

because again like we’ve been doubling with uh automation a lot so we uh

because uh basically when you are automating you can scan you know tens of thousands of firmware so that’s really

good um I I think uh the biggest security mishap that happened with The Meta not

not specifically vulnerability but mishap is uh number one is the lack of mitigation

like 99.9 you know don’t enable stack Henry et cetera and then you find them

the simplest uh stack Overflow and you know you got shell uh the second is

default password I don’t like it because it’s too easy but that’s uh pretty much

uh like the second biggest uh I think third that’s the easiest to exploit I would say command injection

like everybody everybody you know because you’re using C you’re just running P open or whatever you’re not

using a high level language people are trying to filter Bad characters it just doesn’t work

and if it doesn’t have to do the what’s my favorite

and filter for parameter injection so you can inject some flag that uh causes

like code execution like there’s a lot of flags for like tar and git that if you can add a flag you can run code uh

so these ones I really like because it’s like filtered but not really I have a very cool one for automotive so many

Automotive Systems have some form of limited read and write access to RAM and

what is very common is that this is done using a blacklist and not a white list so whenever there’s a new thread or task

or a new feature by default it will be readable and writable in Ram so I’ve seen over the last couple years multiple

times products where they add a new feature they forget to protect it and you can just write and that usually when

there’s any other mitigation simply not interesting because you know Global memory you know where things are it’s

embedded just right there read it figure out what you need to do and you can exploit any device which is really crazy

and this is actually quite common any more questions from the audience

okay so we have a bonus question uh this one is for yes which iot devices will never enter your

home and why

I don’t like stuff that has like microphone and maybe even some kind of

Health sensors and I think the system where all of this comes together plus no

content is baby monitors so there are those Those sensors that you can put on

the feet of your baby so it monitors like the heart rate and blood oxygen and

then combined with some camera and loudspeakers so that like when the child is waking up at night you can’t care for

the child so this is like really creepy stuff and it’s also it’s Health Data

that is collected and if the child has some issue like some higher issue or

something this will be like with the security of today stored on some server

and who knows if it’s going to be hacked in the next 20 30 years

So my answer is also baby monitor but I’ll find uh I can also say

something else but uh We’ve also we actually reversed quite like a ton of cameras and uh some of them were even

specifically baby monitors and the results were not good for the vendors let’s say and like you said there’s a

lot of low-cost ones uh which like yeah no way not after what I saw I I

would say the second thing is uh smart ovens and smart kitchen appliances because

the implication of course is uh if it gets hacked you know you can do it what do you care about your house being burnt

down don’t you have like an insurance or something yeah maybe it would be nice for uh

insurance fraud or something oh it’s spontaneously combusted um but uh yeah that would be my number

too but I have to agree with you on uh baby monitor Yeah Carl do you have any device you wouldn’t have in your house

so I don’t have anything specific but I think that anything with some microphone on a camera will not enter my house

um generally I think that everything in the end could be hacked so it’s kind of

you choose what will be exposed and this is something that I would just wouldn’t want to expose

um in my house so I would say a smart watch with a microphone and GPS I have a smart watch

here but it only has Bluetooth and the reason is that these devices are such small form factor they only have

embedded cores they have the worst mitigations possible it would be a Linux device maybe it would be a bit safer but

once you go to really like deep embedded there’s no chance to secure this correctly

um okay so I think we’re basically done with the panel for today I hope you all enjoyed it a personal note for me my

grandfather passed away about a month ago and he was an electrical technical engineer but actually his business was

uh inventing and designing stationary and drawing equipment in England and he became one of the largest sellers of

this kind of equipment he built his business from scratch and he was always a very positive engineer he always

looked at how technology could get us further and how we could make the world a better place using technology now the

people sitting here today are all security Engineers or first Engineers analysts I I would like you all to take

it to your daily job not only you know we all want to make money of course but how your work can affect and make the

world a better place and I think when we talk about embedded security that’s you know a lot of safety critical stuff that’s one of the things we’re all

focused on regardless of how we’re working on this so with that thank you very much and that was our panel

[Applause]