Road to CTA SalesforceSaturday brings Salesforce architects of all levels together twice a month to connect virtually on their journey towards the vaunted Salesforce Certified Technical Architect (CTA). They invited Waqas Nazir, DigitSec’s founder, to join them and talk about Salesforce security considerations. Watch the full video above or read the full transcript below!
In the conversation, Waqas reveals that he is passionate about addressing the many different security vulnerabilities within the Salesforce ecosystem in a holistic way. From user roles to customizations to test environments, he touches on multiple points related to keeping your Salesforce Org secure from vulnerabilities.
Salesforce has a strong Identity and Access control foundation that architects can build upon. Carefully considering the different roles your users might play as you direct the flow of data through your organization is a key step to a secure design. When new users join your Org, user roles will change. Therefore, the design of that process must be able to evolve and change with your needs.
Customizations on top of the platform are often responsible for the greatest Salesforce security vulnerabilities. There aren’t a lot of built-in guardrails to alert architects, developers, and admins to security issues. It is critical to address security before those customizations are put into testing and production. When key functionality runs with Systems context, it can undo all the work invested in a solid data-security model. Architects need to be mindful of the implementation paths for their designs. A mistake in execution can lead to security vulnerabilities.
One of the first steps for good security design Waqas endorses is the principle of “least privilege” as a base layer of the Salesforce Architecture Pyramid. Basically, start with the most restrictive data policy and then only lift restrictions as requirements demand.
Salesforce is a web-based platform. On top of regular application security considerations, like when sensitive data is exposed and to which users, architects and developers need to also consider browser and web-based attacks like SOQL Injection, Cross-Site Scripting, and Cross Site Request Forging.
Waqas states that special consideration should be given to the design of Salesforce Flow chains. You want to make sure that the permissions across the entire chain align with users that have access to trigger the flow at any point.
The key is to make security a process. Taking the time and making the effort to secure a Salesforce org cannot be labor and time intensive. Change is a constant in the Salesforce ecosystem and teams need tools and processes that they can rely on to check quickly and frequently. Salesforce is a system that is metadata driven. That architecture is beneficial to the security process because you can rely on it to build reporting and track changes in your Org over time.
Other important points Waqas mentions include:
- Dev/Test environments that are used for Continuous Innovation/Continuous Development must be as secure as Production. Salesforce Dev/Test Orgs are publicly available on the Web and are therefore vulnerable to exploitation that could later be pushed into production.
- Salesforce’s holistic design centralizes the management of system audits, encryption and identity management thereby uniting Development and Operations and giving teams a great opportunity to do security the right way from the start and into the future.
Executives and C-Suiters should not rely on Salesforce to do all the heavy-lifting on security because it is a SaaS platform. The Shared Responsibility Model means the customer must do their part. Even small projects should be designed with best practices in mind, particularly developing documentation and processes for managing change as it evolves.
In conclusion, Salesforce security to prevent vulnerabilities needs to be an ongoing process. It needs to be a state of mind. Ideally, everybody on the team has the ability to look at the changes they are making to the Salesforce environment from a security perspective. Security is a process, not a panacea.
Read the full transcript of the conversation below!
Meighan Brodkey:
All right. So why don’t we get started with some intros now that it’s 10 after? I think we had a nice little break. I think we are giving some people some time to join us. Thank you everybody. For those of you who it’s your first time joining our SalesforceSaturday Road to CTA and beyond, starting SalesforceSaturday for anyone interested in the Salesforce platform and connecting with Salesforce, learning more about architecture, learning more about Salesforce, prepping for your pyramid, as well as prepping for your Salesforce boards, and just getting more information about Salesforce, I’m one of your co-leaders, Meighan Brodkey, and then we also have Paul and Daniel.
Meighan Brodkey:
Hi. I’m Meighan Brodkey. @MeighanSF on Twitter. Been in the ecosystem 14 years, going on 15. And if you name it with Salesforce, I probably do it. I’m a technical architect and practice manager over at Xede Consulting and I’ve been there for about two and a half years. Also a user group leader, podcaster, and I’m working on studying for my CTA. And I drank that Kool-Aid and then I didn’t just keep drinking on the Kool-Aid, I just jumped straight into the deep end with it and never got out of the pool. Paul, you want to go and give a high level intro of yourself?
Paul McCollum:
Sure. Hey, I’m Paul McCollum, an enterprise architect for about the past 20 years. I’ve been a developer for the past 40. Done a lot of work in retail, started out as a WAN architect in Nortel in the early thousands. So started drinking the Kool-Aid of… Gosh, was it 2017? I entered and won a development competition for Salesforce and started putting all my interest over to the platform and the ecosystem and how well it was growing and how fun it was to work in. So, that’s me.
Daniel Gordon:
Hello everyone. Daniel Gordon. I’m a Salesforce solution architect in the nonprofit space. I’ve been loving this platform and community for over five years. I learn so much every day and I love doing what I can to support the community through this group. I also co-lead with Jade and Antonela, the Boston Salesforce Saturday group, which is coming up next weekend. And yeah, just welcome and thank you all for being here.
Meighan Brodkey:
Excellent. And then just to start introducing our special guests. So we have a couple of people here to talk about security. We’ll start with Shannon and Ryan and then we’ll launch our guest of honor, Waqas. So Shannon, why don’t you start?
Shannon Smith:
I’m Shannon Smith, COO at DigitSec. Like Paul, I come from way back in the day, grabbing a keyboard as soon as I could write basically, and love computers, the whole stack layers one through seven. I’ve done it all a little bit, but yeah, one of the areas I’ve found most interesting is security. I’ve been working professionally in cyber security for last 20 years. And yeah, I started using Salesforce around 2005 and doing some power using, I guess, early on, just because I never had an administrator available to me, but certainly I’m an amateur compared to any of the folks on this call. Nowhere near an architect. Like an architect in my mind, but that’s about it. But yeah, our company is providing a full spectrum application security solution that’s purpose-built for Salesforce. Nothing like anybody’s ever seen before, and we’re real happy to be here. Thanks for inviting us.
Ryan:
Yeah, and I’m Ryan. I’m actually Shannon’s younger brother. I work with Waqas and Shannon at DigitSec, and I always love joining these because I love hearing the questions people ask about Salesforce Security, and every time Waqas speaks about things, I always end up learning something myself. So, happy to be here. Thanks to Meighan and Paul and everyone for setting this up and doing this on Saturday. Appreciate it.
Meighan Brodkey:
And then Waqas, our guest of honor. Security expert. Go for it, man.
Waqas Nazir:
Thank you, Meighan. My name is Waqas Nazir. I am the founder and CEO of DigitSec. We are a security testing platform for developer security for Salesforce. My background is primarily in application security starting off, and I pivoted to Salesforce once I was doing a lot of assessments for ISVs and also a lot of pen tests for Salesforce customers, which excited me to this platform in the sense that there was a lot of moving pieces and security was being implemented in many different places within the platform. So looking at it holistically was very interesting to me. So I ended up creating the product to address that particular need. So I’m happy to be here and talking to you about a field that I’m quite passionate about, and thank you for having me. And I hope it’s a useful session for you all so you can understand the security of the platform better, but also look at security from maybe a attacker’s perspective, which I believe is very valuable when you’re trying to secure any system.
Meighan Brodkey:
Awesome. Thank you. All right, so let’s just jump straight on in. So Waqas, so you mentioned taking a look at security from a hacker’s perspective, and I definitely want to get to that, but I also want to start with what are some of the general things with security that are out there about Salesforce that are some of those best practices that people should be paying attention to and should be following with their build. Some of those foundational items that people should be keeping in mind, especially when solutioning end-to-end testing. So on the solutioning side, what are some of those items of security that we should be keeping in mind?
Waqas Nazir:
That’s a very good question. So designing on Salesforce, I think requires a bit of concentrated effort if you want to design it securely from the ground up, and it starts with the base of how the data will be accessed. So I think the key foundation for Salesforce as a platform is identity and access control, and I think that is the core of where the design should happen from a security perspective. So really understanding the problem that you’re trying to solve, or the business function that you’re enabling and understanding all the different types of users you’ll have, all the different personas and use cases, and coupling that with the type of data that you’ll be dealing with so you can design your identity as well as your access controls properly, is I think the baseline for designing a secure implementation of Salesforce, and from experience that is an area which is often the last, or the thing that evolves as people start using Salesforce.
Waqas Nazir:
So as an architect or somebody who’s implementing a solution and having that foresight and how this will grow, how the platform will grow, how identities will change, how new users’ cases will be supported, and how you can make sure that the data doesn’t get exposed, there’s all elements of this base that you want to create for Salesforce. So I think the number one thing is creating the right data model and access control and understanding all the different pieces that go with that in Salesforce is critical. One of the principles that I recommend from a security perspective when doing that is the principle of least privilege, and that’s implemented in Salesforce that pyramid way. So you start with the most restrictive data access policy, which is the least data policy required for somebody to perform their action, and then you open up based on requirements to enable more access.
Waqas Nazir:
But when you start enabling more access, that is also an area where there’s a lot of pitfalls and things like that you have to be aware of. Things like those are more implementation-specific, but you should be aware of those as an architect because if for every single thing that a person has to do, you have to create a new permission set or keep extending profiles or sharing rules, then at a point, that may become something that is not manageable and things like that. So understanding all those aspects and creating a data model that can scale and be secure is critical from a design perspective. So I think that’s one thing. From the basics, I would say that’s the most important in Salesforce.
Meighan Brodkey:
Nice. And I think we all know what some of those side effects are that happen when you go for too much access. People end up being able to touch stuff they’re not supposed to touch. People end up somehow getting into the set up and moving stuff around when they’re not supposed to, and then should you fix it themselves, or they just end up doing things that aren’t their job. And next thing you know, you’ve got stuff that they weren’t trained on happening, or you’re opening up your org to entirely new issues that don’t even relate to the people inside your org. Where do you see it, some of those most common security issues in Salesforce happening, and what are some of them?
Waqas Nazir:
Sure. So I think one thing that, based on what I was just sharing, a lot of things, like you said, that we observe are around data access. But also where we see there’s a lot of attack vector and there is an area that is often overlooked is customizations on top of the platform. So even if you have a great data model, you have great provisioning permissioning and all that in place, when you start customizing Salesforce by writing custom code or installing third-party components and extending Salesforce beyond its core box by connecting apps and enabling remote sites and all of those different features, each of those have a security implication. And those are where there’s not a lot of controls in place to assess these changes before they’re put in production. And we see like when you start adding custom code or adding custom things to Salesforce, if you don’t address the security of that, you’re basically opening yourself up to attacks.
Waqas Nazir:
These attacks are basically any attack that you will have to like a common web application, because in the end of the day, Salesforce is a web-based SAAS platform. So things like SOQL injection, things like authorization bypass, things like cross-site scripting, things like cross-site request forging, all those well-known web exploits, and then things specific to Salesforce. Things like having sensitive information in your debug logs or having sensitive information in your view state, stuff like that which are specific to the Salesforce platform. All those attack vectors, if they’re not addressed, they’re for somebody to export. So that’s an area that we see is, as Salesforce grows and has all these tentacles going out, enabling all these features, we see that those attack vectors are then things that lead to potential data breaches and people hacking and changing stuff that they shouldn’t be able to change.
Waqas Nazir:
We’ve seen, for example, Salesforce is driving sales, and there’s orders and there’s pricing and codes. So we’ve seen exploits affecting hundreds of millions of dollars worth of orders. So it’s not something trivial, but sometimes people think that since we’ve got our permissions lockdown, since we got good admins, we have a good workflow around identity and access management, we’ve got it squared. But Salesforce’s custom code runs with system context. So those things can basically trump all the effort put in by a good architect or admin to maintain good data security. So from the common things that we see, or which are overlooked, that’s one key area, and Salesforce keeps coming up with new features like Salesforce flows and things like that, which enable more data access, and then they have the ability to circumvent the data model.
Waqas Nazir:
For example, what’s very interesting with flows is, just to give you an example, one is the obvious thing that you can mark the flow to run with system context, which is basically the same as giving somebody modify all data, like the MAD permission. So that flow basically runs with MAD permission. But what’s also interesting is that Salesforce flows can be chained. You can chain different flows together. So if the user has access to trigger the first flow, and if the second flow runs with system permissions, and the users should not be able to run that flow, they can trigger that flow by triggering the initial flow that they have access to, which is very interesting. So security doesn’t flow in the flow, if you understand what I’m saying. So the context doesn’t flow. So those are some of the things that we see that often are overlooked and can cause serious vulnerabilities and harm the Salesforce work.
Paul McCollum:
So if you’ll let me, I’ve got a question. A big subscriber to least privilege. I have trouble moving from project to project or going in and out of orgs and not being able to understand their permission security structure. Have you seen any… I don’t know if there’s tools. I hope I’m not just setting this up as a softball question, but for understanding and visualizing security model, when I work on projects, so like, yeah, grant this person access to this profile, or grant this profile this permission set. And I’m always stuck, well, how do we know they don’t already have permission through some other means with all the overlapping ways to get access to objects? There’s a role hierarchy, you can set that up. And that’s very visible and very seeable, but all of the permission set seems to be one-off to each permissible, I don’t know if that term has been coined yet, but how do we visualize and understand org security?
Waqas Nazir:
That’s a really good question. So, part of the problem, I think is, like you said, there are so many layers to where somebody has access, and it’s hard to decipher those. We, as a product, have this feature where you can actually look at cumulative permissions by objects from an attacker’s perspective. So if you have users who have excessive permissions that they shouldn’t have, those are the things that we can flag. But I think visualizing the current security model is I think a challenge, and it is something, I think, that could be automated, and that will be a lot of value. But to your point, there isn’t an easy way out of the platform to understand holistically like, if I choose this user and this object and these fields, tell me how this user has access to this. So that a path can be visualized easily.
Waqas Nazir:
So I think that’s an interesting scenario to solve from just a project management perspective. So I would say there’s a way of doing that programmatically, and we have those tooling because we do look at cumulative access right, and you could technically extend that for everything. We’re looking at it from an attacker’s perspective. Why do you have somebody with a view-all-data access while they are lowest part in your hierarchy and things like that. So those things we can look at, but I think to your question, there is an opportunity to perhaps visualize it for across the board so that it’s easier to understand those things.
Paul McCollum:
Okay. Thanks. Have you seen anybody come up with the interesting patterns on managing and maintaining their org security that helped them prevent attacks or helps them manage and knowledge of their org security that keeps them out of the cross hairs or…
Waqas Nazir:
Yeah. Not to sound like the product pitch here, but DigitSec S4 actually helps with that. So we look at Salesforce holistically. So we look at all the custom code that’s deployed within the Salesforce, and I think that’s the best practice regardless of how you want to go about it. That you look at all the moving pieces within the Salesforce. So one is your identity management and access provisioning. That’s number one. Second is your custom code and customizations. Third is, if you rely on any third-party components and you rely on open-source software or what have you that’s part of your org, how are you keeping that up to date? And then the fourth critical element is basically looking at the environment, how it’s configured. So basic configuration settings. If you are able to look at these key elements, then I think you’ll have a pretty good chance of keeping your org secure.
Waqas Nazir:
The other thing is, you may do this exercise once. You go through this exercise and you are able to get to a level of assurance that, yeah, we’re doing the right things, but tomorrow a change can happen. There’s constant change happening. There is admins changing things. There are developers deploying code. So to maintain that, you need to have some tooling around it and some process around it to make sure the tools do get run, and then the issues that the tools identify or flag are able to be triaged and addressed. So it’s not just a check box, there’s actually security coming out of it. So those I think are the key elements of making sure that your words are secure. Salesforce does provide basic configuration checks for the health check. That’s always a good practice to do, but that again is like a point in time check. It’s not something you can do every single day, go click on that and see what has changed. So you need some automation to address that, but there are some tools that can help you, at least on the config side, to do that.
Daniel Gordon:
And what recommendations do you have around documentation? This has come up in a couple of other meetings where… Paul, you were talking about coming into orgs and… Just any general tips on ways to lay that out in a document so that someone coming in to the org next time, or just even for your own, or if you’re working with the team, everyone’s on the same page about the security landscape from the profile to the different apps that are installed?
Meighan Brodkey:
Oh, good. Especially since as soon as you make that document, it seems to be out of date. How do you keep it up to date?
Waqas Nazir:
Right. That’s what I’m saying is like, anything that you do in a static manner, it’s bound to become a hassle to maintain. So you really need some tooling around the current posture of your environment. So let’s say you’ve approved a list of apps. You’ve gone through and you’ve approved them, but those are permissions who are able, so they can add new apps. So are you going to go and update Excel spreadsheet or Visio document to do this every month or something? It’s pretty hard to do. So you really need some automation around looking at Salesforce holistically and telling you that a, you went through this process and you identified this is the way your work should run from a security standpoint. These are the permission sets, this is how we’re going to configure them. But now this is what has changed. So you need a process to almost give you a security org assessment which is live and it doesn’t go out of date. And it gives you all the moving pieces that, hey, here are all the remote sites that you’ve allowed, but somebody just added a new one.
Waqas Nazir:
And did you know who did that, and why did they do that? So all that stuff, I think, if you can build automation on top of that to alert you on those things versus having to find out that you left your Salesforce on this corner, and it has climbed up the stairs and gone somewhere else now. So DigitSec S4 enables some of that automation. So it can sense changes and alert people that, hey, this has changed in your environment. Do you really approve this change or not? So it gives you a history of how things were and where they’ve gone. And I think that is where we would recommend people to go instead of creating documentation that becomes hard to maintain. And I think that’s the beauty of Salesforce is that it is metadata-driven, so you can build a lot of reporting on top of it using the metadata, and making sure that those changes are tracked within the environment.
Waqas Nazir:
And then there’s, of course, CI/CD tools, which enable you to document those changes and application life management tools that you can employ as a process to make sure that no change goes into production, all changes happen in the sandbox from before that. So everything gets documented, and once in production, there are actually no surprises for an admin that, Hey, who did this? There was an interesting meme going around about Salesforce environments back in the day and how they’ve progressed over time. And for some, I think, for not all, but there was like, who made this change? Why did they make this change? And then there was somebody yelling that, “Does anybody know what’s going on?” so that is a worst case scenario for any admin or architect because they have no idea of who’s making the changes, why are they making them, and what’s the current posture of the environment. So that would be my recommendation instead of managing static documents.
Meighan Brodkey:
So I wrote two questions down in the chat so I didn’t forget them. Speaking of CI/CD, how do you ensure your CI/CD process is secure? So going beyond your org, that process to migrate things. And then I’ve got a follow-up question. It’s not related.
Waqas Nazir:
Sure. Good security practices within the CI/CD pipeline is a must. We’ve seen, unfortunately, that has become an interesting attack vector, because like the SolarWinds hack, which basically altered code within a CI/CD pipeline and that impacted thousands of customers across the board. So interestingly with Salesforce, typically CI/CD pipelines used to be in the back of an enterprise. Only developers had access to it, and it was kind of a black box. So once people shipped software out of that black box, they had some controls around making sure that software is secure. That’s not to say that black box can’t be compromised, but it was still few layers of compromises you needed to get to that box. Now, especially with Salesforce, all the dev orgs, all the Sandboxes, they are actually public, right? That’s by design, because you’re deploying to them, you’re looking at changes from them. So for Salesforce actually, CI/CD pipeline is actually more attractive from a hacker’s perspective.
Waqas Nazir:
Because, that’s where people think that, “Oh, it’s only dev. I could have LAX credentials, I don’t need an entity access management.” Because it’s just dev, it’s test. “Oh I don’t need to create stricter controls around data access because we’re just using a subset of the data.” So I think it is important to address security across the board, and with Salesforce, especially since it’s all the … available on the internet and it’s available … that it can compromise the security of your environment by being exploited in dev or sandbox.
Waqas Nazir:
So I think it’s really important to see those things in the CI/CD pipeline and secure the CI/CD pipeline.
Meighan Brodkey:
Awesome. And then follow up question totally unrelated. What are some of the most impressive features in your mind or some of the best features out there for security on the Salesforce platform?
Waqas Nazir:
So from a security perspective, I think the fact that the platform has really good auditing capabilities, right? The ability to look at changes and be able to trace those changes, is actually a really good feature. The native features that enable like encryption and … basically the ability to manage identities across different systems, is also a great feature. And if implemented correctly, can provide a lot of value. So those are some of the things that I find are really beneficial and great in Salesforce.
Waqas Nazir:
The other thing is, since it’s a holistic platform, it also gives you the opportunity to secure all the different elements in one place. Because, there’s sometimes a disconnect between development and operations, but you can bring them together in Salesforce, and really secure our system well because, all the different components are in this one platform and this unity brings this great opportunity of doing security right.
Meighan Brodkey:
Nice. Awesome, thank you. I know we’ve got some questions in the chat. Paul, you want to start taking those things?
Paul McCollum:
Yeah. I was just about to hop on that. So, let’s see. Yep. I’m not muted. Mary asked, how does anonymizing the data in Sandbox help security?
Waqas Nazir:
It definitely helps the overall security posture, right? Because, you don’t have the connection with actual customers per se, right? It’s still PII data. If it’s PII data, even though it’s not connected to a customer’s instance, a particular customer, it’s still valuable. I think anonymizing it takes away the opportunity for a hacker to actually make it more valuable. It’s still valuable, right? From a security perspective, if you compromise an anonymized data set, which can be sold on the dark web, it’s still valuable, but it’s not connected to someone.
Waqas Nazir:
So, its a value is not as important, but let’s say if you have anonymized credit card numbers, it’s still pretty valuable, and it’s something that can be used. So, anonymizing is good because it takes away the connection of that data to a actual customer experience or something like that. But from a security perspective, you really want to protect that data. You don’t want that data to leave your environment or be accessible to someone. So I think for those, if you have sensitive data, then I think proper controls around that should happen. For example, if it needs to be encrypted, it should be encrypted.
Waqas Nazir:
If is highly sensitive, it should not be accessible to everybody. And then I think it’s always a good practice to have a test data set, which has the same qualities, but not the same data. Right? So if there are, you could use test credit cards for testing, you could use test SSNs for testing SSN stuff. So things like that I think are better approach to securing your data than just anonymizing data for sensitive data.
Paul McCollum:
Then I hope you weren’t expecting a structured pattern of questions. We’re all over the place, hitting you from all sides. So, thanks for fielding all these. Next question, what do you consider the most vulnerable surface in the Salesforce ecosystem that users are overlooking? Where are you coming up and setting? Where do you most commonly see breaches that need to be mitigated?
Waqas Nazir:
Very good question, and I don’t know if somebody is on this call who got an alert from Salesforce recently to secure their Community site, but that’s a very low hanging fruit for hackers and malicious users or bug hunters, which is basically, querying your Salesforce instance and say, “Hey, give me all the account objects.” And if you have that enabled, Salesforce will happily give you all the account objects. If you have contracts, somebody queries, you have that enabled for guest users, Salesforce will happily give you because you’ve configured it that way. So I think that’s a very easy exploit for anybody wanting to do harm.
Waqas Nazir:
This is why Salesforce sends out this alerts saying, “Hey, do you really want to expose these objects on the internet?” And that’s one area that we’ve seen, which is pretty common and it’s exploited quite often. The other interesting attack vectors from our perspective also is, a lot of anonymous features like Web To Case, Web To Lead are in actually a very interesting attack rector, because you can have a Salesforce instance which is in a controlled environment. Everybody has an identity. Maybe you don’t have any guest users and maybe you don’t have any external users, partner users.
Waqas Nazir:
But you do allow some outside customers to interact with Salesforce by creating leads or cases and things like that. So you always have this anonymous user, the ability to inject bad code or bad data within your environment, which could be code. So we have an interesting exploit that we drafted which basically allows somebody to inject stuff from a contact us form, which creates a Web To Lead. And then we exploite across a scripting issue, and then we escalate privileges. So that’s an interesting thing that we’ve seen.
Waqas Nazir:
It’s just based on code that we’ve actually observed in the ecosystem. So those are, I think some of the low hanging exploit vectors. If you look at it from an internal user standpoint, then there’s many. People with access, have many URL hacking and just brute forcing IDs and getting data, all that stuff is fair game for an internal malicious user. So those are some of the things that I can share. There’s many more.
Meighan Brodkey:
A really quick just side notes. I know Salesforce keeps putting a lot of stuff out there to help prevent sharing with … you sharing of and, around a new guest rules, a lot of new preventions for putting and exposing too much data to community users, but people I’ve seen keep trying to get around that. And I know Nate called us out the other day as said that with account name is not equal to blank as a sharing rule and things like that, these rules are going into place for a reason to help you secure your data, Salesforce isn’t doing them just because. So definitely, please, try to pay attention to them. There’s a reason behind them. So, yeah.
Waqas Nazir:
Yeah. And I think with this new release of enforcing guest user policy, they have taken away edit and update on records created by guest users. I think that’s a really good step in the right direction. But like you said, when you create a sharing rule for a guest user or an external user, and you create rules like nonsensical rules like that, that basically exposes all your data, all records, right? We’ve seen, for example, people create ridiculous rules like that, where name is only an integer type. Then don’t share that. Names are typically all is contained in alphabet.
Waqas Nazir:
So, all that data would get exposed. So I think these are rules and ID [inaudible 00:40:40] basically means that you want all of your data for that object to be accessible to all people in the world, on the internet. And so, although Salesforce is trying to be the best to give you the right tools, people can still make mistakes or on purpose just because it’s hard to test or it’s a mess and you just want to give access to everything because that’s the only solution that you can come up with, actually that has serious security repercussions.
Waqas Nazir:
Because at that point, it’s basically the same as giving view all access to that profile. So thanks for mentioning that, Meghan.
Paul McCollum:
Jade had a question. Are there strategies to convince an organization’s C-level execs who don’t really believe in security to take it more seriously? Throwing out acronyms like PHI and HIPAA and GDPR, any other strategies to encourage them to hire strategically?
Waqas Nazir:
That’s a great question. Now I’ll invite maybe Shannon to chime in, but from a security perspective, we’ve seen that, on the C-evel, Salesforce sometimes is operating in a very controlled environment that’s driven by sales primarily. And it is a customer relationship management system. So, the driving force for changes in the environment are sometimes even overlooked by executives. Because they see that it’s a SaaS platform. Salesforce takes care of all the security and the reason we use it so that we don’t have to worry about security. So for C-level organization, I think it’s important to create an understanding that there are only few things that Salesforce is taking care of from a security standpoint.
Waqas Nazir:
And there’s a whole lot that you as a customer of Salesforce are responsible for. And it’s setting pretty good terms on Salesforce inside that security is a shared responsibility, which basically means that, we will do our part, but you have to do your part. And if you don’t do your part, and you end up exposing your data, we’re not really responsible or liable in any shape or form. And they give you the tools to make sure that you’re secure, that’s their responsibility. So I think that’s the number one I think argument that you can present at the C-level, to make sure that they take security seriously when they’re operating in the Salesforce environment. But Shannon, feel free to share some insights on this as well.
Shannon Smith:
Jade, great question. I ask myself this every night before I go to bed and every morning when I wake up so, I’m actively striving for the perfect answer to this question. But yes, first of all, those acronyms, compliance, motivates budget, and budget after being in the pen testing world for 20 years, you scrape and beg for a budget to do a $50,000 pen tests sometimes at certain companies. But, you want to do a compliance review, they’ll chunk out three, four, $500,000 without thinking about it. Because that’s basically compliance allows them to do deals with big customers, right?
Shannon Smith:
Compliance, if you don’t get it done, stops the revenue flow. It’s one of the reasons Salesforce security has been an afterthought because the main focus of Salesforce is accelerating the sales process, right? And making it more efficient to get all that customer data to all your salespeople out in the field, and everyone else that’s involved in that. And so there’s this kind of fight between access and speed versus security and safety, right? So, compliance can help motivate that for sure, and that’s a great one. In Salesforce when it’s not properly maintained, does violate compliance.
Shannon Smith:
The amazing thing about Salesforce so once they start using it, and it’s super secure day one, and it has all these compliance logos all over it, and it’s great. But as you modify it, those logos, those compliance frameworks stretch out farther and harder to reach and then of course your actual security posture and the chance of you losing your customer data increases over time. So yeah, anything you can do to wake the executives up, at the end of the day, Salesforce pretty much contains the most valuable information in your company, which is all of your customer information, all of your deal flow, so if you’re not getting proper support, I say you corner the CEO sometime at the random coffee or water stand once we get to see people again and ask him how secure he or she thinks Salesforce is.
Shannon Smith:
How secure is the Salesforce? I work on it every day, but we don’t have a huge budget for security. Have we done an assessment, right? Do we have an idea? What’s our posture? So, ask those questions people get thinking about it, then they start working together. There’s only like one security person in a big company sometimes two, right? And so, it really it’s to all of us to be security people. And we got to pull on the execs for that support and that budget, and that training and all that good stuff.
Paul McCollum:
Yeah. Mary is bringing up a good point on the call. And I actually heard about this through grapevine where somebody printed up a copy of the fines that another company had received based on not complying with certain orders. This was more around, I think the unsubbing to email lists and stuff like that. Like, here’s the standard set of governance fines. You probably want to go ahead and find this in your budget right now, if you choose to not follow these. So yeah, I think CCPA has a listing for punitive damages and so forth like that. So.
Shannon Smith:
It’s funny, you mentioned CCPA too because the first company to be sued on January 2nd, 2020 by CCPA was Salesforce. So that was the very first violator of CCPA, a company which has a great record on security overall over the last 20 years. So Hannah Anderson, who makes children’s clothing, got their stuff hacked in Salesforce and there were class actions, lawsuit from customers and from the state of California first day of January last year. I gave you the more diplomatic answer, right? When and how to motivate executives to focus on security and give you budget and training and all that good stuff.
Shannon Smith:
But you can also just throw this hearsay out there, which is, “I’ve seen a lot of security audits in the last 10 years in the Salesforce arena. And I pretty much have never seen one that has …” There are some rare cases, but pretty much all of them have high risk vulnerabilities, whether it’s a dev situation or just an admin, unless you’re just using Salesforce out of the box, Salesforce is so amazing. Like you can do anything with it. It’s like the ultimate play dough or 3D Printer right? You can just create … it’s like a Minecraft, right? But the more you create access to stuff, that’s all attack surface, right?
Shannon Smith:
And the general production Salesforce org that we scan has thousands of vulnerabilities in it. If it’s been in use for over five years and people have been doing stuff to it besides just using it, it’s most likely got hundreds, if not thousands of vulnerabilities in there. So if you want to use a little FUD I mean, that’s pretty much … we’re going to publish some stats later this year, but there’s a lot of … all that access is also attack surface.
Paul McCollum:
Is there a zero day listing for Salesforce or is that something that haven’t even peeked into to Salesforce’s generic public security concerns.
Shannon Smith:
Thats a good question. One thing also, I’ll just throw out there is that Waqas has a demo hack. We can show if anybody believes Salesforce is just absolutely secure and can never be cracked, where we leverage two vulnerabilities and go from contact us page to admin of your Salesforce in less than five minutes. So we can show that if we have time later today.
Paul McCollum:
Yeah, no. That demo’s amazing. I’m hoping we get time to show it.
Waqas Nazir:
So yeah. To your question, I think there’s been like one large scale incident with Salesforce, right? There was a database script that expose data for multiple customers, which was a faulty database script on Salesforce side. So there’s not a whole lot of zero day activity happening on the Salesforce side, but the technologies that Salesforce relies on, of course, has many bugs that are impacted to Salesforce, but that’s not in the purview of public knowledge. Because it’s a closed system from a user’s perspective. So, I don’t think there is outside of that database script, any non-zero days for Salesforce.
Paul McCollum:
Now, if your security admin refers to it as an “Ohh Day,” do you fire them or do you just lay them off?
Shannon Smith:
Ha ha ha! And you don’t lay any security people off! It’s so hard to find them. You educate them and try to ask why they are mistaken. That’s my first recommendation.
Paul McCollum:
Yeah. That was my least favorite conversation with the security professionals. And I I was hoping, I’m glad to hear you guys say it as “zero day”. It’s one thing you type everywhere, but like, “Ohh Day” what’s an “Ohh Day” attack? Heh Heh. What are you talking about? Oh, you mean zero day. Yeah. red flag.
Waqas Nazir:
Typically zero days… It’s really been many days and many people have already been exploited by it. It’s technically not even-zero day.
Paul McCollum:
Yeah. Fun stuff. Are there any useful blog sites or places that you go that where people who are doing research on Salesforce security and posting news, where can we go subscribe? What feeds can we monitor? And again, I’m not intentionally soft balling you all of these questions for you to advertise, these are legitimate. Where should people go and read to make sure they’re up on the latest security goings on with Salesforce?
Waqas Nazir:
Yeah, I think the great resources is the Salesforce security hub, I think it’s called a Cybersecurity Hub, where it allows the different personas to be able to understand cybersecurity from a Salesforce perspective, but also talks about a lot of compliance related initiatives within Salesforce. Another great resource is the developer guide, the secure developer guide, which is on, I believe the Trust.salesforce.com site, which basically trains developers on, what are the security implications of code in Salesforce, so writing Apex, Lightning web components and the original force pages, things like that. Those are, I think some of the good resources that you can rely on. Also Salesforce Ben has some great articles as well.
Paul McCollum:
Oh, Okay. That’s a good question Meghan, do you want to ask, or you want me to ask?
Meighan:
Go for it! I’ll grab any other questions we have from above.
Paul McCollum:
Okay. So from a CTA perspective, at what particular point do we need to maintain a checklist guide of best security measures taken within an org? There various SMEs that have Salesforce implemented and most of them are sort of quick start projects, but if we focus too much on security, the cost of the implementation goes up. So is there any guide or formula to consider to spend time or efforts solely on security implementation? I think that’s basically asking what’s the ratio of project hours to security dev that you think we should see or invest on new implementations?
Waqas Nazir:
Yeah. I think it’s a great question because the whole point of having a platform like Salesforce is starting easily as a low barrier to entry and then customizing it as you need. But I think from a security perspective, not having any idea of how you’re going to manage personas and permission sets and things like that, right off the bat, and then thinking, “Well, we’ll figure this out six months down the road.” Will be very problematic. So having at least some semblance of how you’re going to manage data access and identity on the get-go, I think is a recommended best practice.
Waqas Nazir:
Then also create a process on how you will change these things. And how you will manage the security of adding new code or enabling new features or when we’re creating flows, we must always have permissions inherited and not have everything run as system and things like that. So, those best practices I think should be documented from the get-go so that you start. Then I think as things change, as you add new features, then you can start documenting that, yeah, we’ve added this new way of connecting this external system, how do we secure it?
Waqas Nazir:
So I think from a security perspective, whenever a project gets started, I think having a discussion on what is our critical use case, how do we protect against someone malicious? Creating this type of model in your mind. What are the main threats to the data in this environment? And you update that as you go, as you add more features, as you add more … do more implementations and projects to it, having security as a critical part of it, will keep you updated. A good process to manage that is published. For example, the Microsoft SDL process, that talks about all these key elements as you’re doing a project, and it’s designed to be agile.
Waqas Nazir:
So it’s not like it is very cumbersome. So when you start small, have small best practices with that in mind. And then as you grow, you can change and adopt to the needs of the security. So I would definitely recommend looking up the STL process from Microsoft to use as a standard best practice.
Meighan:
I’m really quick to emphasize, document those processes, get them down on paper from the beginning, get a digital copy of what that process is going to be and make sure everyone understands it. The worst thing that you can do for yourself and your team is to be continuously changing the process week after week and have nobody know what their actual process is. And to say you have a process, but they have no rhyme or reason to be changing it time after time, so that nobody knows what they’re actually doing, and then to have actual security mishaps, easily sneak through. Sorry.
Waqas Nazir:
Very good point. Well thank you for making that point. Yeah, document your data model, document everything that you’re supposed to be doing, when you’re managing data access and identity, so that you don’t have issues Waqas Nazir:
down the road where you cannot change things because you started on the wrong foot.
Meighan Brodkey:
And then one of the next questions was about best practices for any recommendations and best practices when using Salesforce big objects. And then what about external objects?
Waqas Nazir:
Yeah. So it’s basically the same recommendations, right, for a data model? External objects basically get data from an external source. Data access should be modeled the same way as you’re doing it for custom objects because they’re, in essence, an extension of custom objects. And the same level of security needs to be implemented for them. And so, having the standard model will help address the needs for these custom objects.
Meighan Brodkey:
And to add fun facts in about external objects, so with your external objects, you have two ways to access them. You can go by user and get individual permissions set on the other side, or you can have integration user and then you get access to the objects. No, but as long as you have access to the objects and there is no per record access, if you need actual per record security, don’t get the integration user and try to come up with something crazy sneaky in Salesforce. It’s a huge pain and it’s not super secure. So think about what you actually need and what your long-term goals are, and if that data is going to need to be for an individual user as a side point.
Waqas Nazir:
Yeah. I think if the identity is going to be managed by the integration, then you need access control on the external side. But yeah, if it’s the user, then you implement the user security. Yep. That’s a good point.
Daniel Gordon:
A question up at the top I put about AppExchange. Now this install, if you think of our phones, “oh, there’s these app stores and they do their security reviews, so I’m good. I just can install it”. But then as we know, a lot of apps hopefully give you the ability to decide, can it access your camera or your photos? Things like that. So similarly in the Salesforce platform, just curious, it’s another area where I think what you’ve been talking about today applies where you have to do your own considerations of what that app can do and should it be allowed to do certain things in the orgs. Just wondering if you had any sort of recommendations to assess the security of an app and not just assume because it’s from the app store and it goes through a security review, then that’s all as an admin or architect or developer that we have to do.
Waqas Nazir:
Oh, that’s a very good question, and I think it’s an interesting challenge, right? Because AppExchange does have a security review process, right? And they review every application that they allow to be in the AppExchange. And I think from somebody who is implementing the solution, it’s closed source, so you don’t have access to the backend source. You only have access to the static components or the front end of the application, right? So it’s basically very limited for somebody to consume and actually assess the security of that app. From somebody who’s consuming the app, I think there’s some best practices that can be implemented to make sure that you’re not exposing yourself. For example, some apps are designed to be used externally, right? Like on community or experience sites? I think those are very high-risk apps. So making sure that you have the right, backend to support those apps and you don’t have a situation where you are opening up more access than you should for that app. But that’s one. The second is making sure that those apps are up to date, right?
Waqas Nazir:
We’ve unfortunately seen that many of these apps go stale. They use a lot of third party components and they can potentially have issues. So those are some of the things that, as an admin, you can look at for apps coming from AppExchange. The other thing is that if the app was requiring you to integrate into a third party, how comfortable you as a company are to integrate with that third party. So assessing the third party security, is that actually an interesting aspect of using an app?
Waqas Nazir:
So, you may be using an app vendor, but that app vendor may be sending your data to three or four other integrations, right? So understanding all that is actually really important when you’re using an app from AppExchange. Just thinking that it’s coming from AppExchange and there’s no security implications will not be the best approach. And off and on, we do see issues with apps and that’s natural, right? Even with your apps on Android or iOS, there are security bugs with them, right? And sometimes they’re very serious. They expose a lot of data for other customers. So that happens with app exchange as well. That’s just the nature of the beast. But I think for the most part, there’s not a whole lot other than the trust that Salesforce has built with a security review process, that these applications have been vetted and they’re secure and safe to use.
Meighan Brodkey:
And one other thing, so to really quick talk about your product for a second. So I did run a DigitSec S4 review on an org, and I got say, there are a number of apps that ended up coming up with issues, be it high or medium. And some of them had been through AppExchange reviews. Some of them hadn’t. And those unofficial SF components had quite a number. And keep in mind, they’re just available out there on GitHub and via the unofficial SF site. And they’re great and they’re handy, but they have some issues with security. So just something I never really thought about, you know? And I didn’t go, “Hey, Salesforce made these. Although they’re not Salesforce, Salesforce, they’re by people at Salesforce”. It wasn’t something that I thought about as much, that they would be such a giant red flag.
Meighan Brodkey:
And I didn’t realize how much my devs were doing to update them and every time we put them into the repo and made them reusable. So, if you’re using those, make sure that you’ve got somebody that does a review and does the tweaks to them and updates their security, and also make sure that you’re checking out the security of anything that you install, be it managed, unmanaged, open source, and that you still take the time to review those items, especially unmanaged, for you can go in and check out the code and check for any loophole.
Waqas Nazir:
Right! Yeah. If you are able to do a deep dive and get some assurance on it, that’s the best thing. But like you said, again, just because you’ve installed an app, a point in time, it doesn’t mean that it’s going to stay secure forever. Like there’s always new exploits coming out. There’s a lot of third party components which are used. There’s a lot of open source things which are used and that’s actually a pretty interesting attack vector too, right? So if you compromise an application, which is used by a few thousand customers, from a hacker’s perspective, it’s just much more valuable than me just exploiting one Salesforce customer, right? So if I find a flaw in one of these apps, then I can replicate that attack across many people who are using that app. So it is actually very important to make sure you’re assessing the security of the list.
Meighan:
Speaking of the hacker side, I mentioned before, I kind of wanted to go back to the idea of looking at Salesforce from a hacker’s point of view. But here, you mentioned that in the beginning, I’d love to hear more about what are some of those ways that we can look at Salesforce from a hacker’s point of view, as we’re building. Seeing as most of us are builders, not hackers, how do we make sure we’re looking at things and that point of view and making sure we’re compensating for what a hacker would do and getting into that mindset?
Waqas Nazir:
Yeah. I think there are few dilemmas, right, for builders versus attackers, right? Or you can say hackers, whatever term you want to use. And there are some advantages that the hackers have and some advantages that the builders have, right? I can enumerate from an attacker’s perspective. So the attacker basically has to exploit one issue, right? So the weakest link, that’s the target. It could be something more complicated or sophisticated, but they basically need one thing to exploit and get in. From a builder perspective, they have to protect all of their entry points, right? So, for a builder, the challenge is making sure that all of the attack vectors are closed off from an attacker. All the attacker has to do is find one. The other is that for a builder, the attack can be something that you know, right?
Waqas Nazir:
If you know that somebody can attack you using a certain way, you can protect against that, right? For an attacker, there’s always ways of finding new things to do, right? So there’s an advantage for an attacker’s standpoint. Now, the advantage for a builder, which the attacker doesn’t have, is control, right? So the attacker is looking for control, whether that’s the weakest point or a new exploit. What a builder has is the ability and the visibility in the system. So if you want to protect yourself, you have to look at everything that you’re building, how somebody would use the same feature maliciously. One way of doing it is consider everything that comes from a user. Be malicious, right? If you’re getting an ID or if you’re getting a string type, or if you’re thinking that a user should only be able to select from a pick list, what if they pick something which is not available in the pick list? What would happen in the code then?
Waqas Nazir:
So assume every single input into the system is malicious, right? And how do you protect against it? And then, building a system around that. So you have the advantage that the attacker is not built to easily identify these things.
Waqas Nazir:
Malicious hackers don’t play by the rules. We as builders and security people play by the rules. So they have a lot of advantages too from an attack perspective. But the one key advantage as builders that we have is the ability to look at the system from the inside and then make sure we secure. We have the visibility to secure all the attack vectors. But it’s a process, right? The other thing is there’s no such thing as a completely 100% secure system or environment because the world is changing.
Waqas Nazir:
The technology is changing. The systems themselves are changing. So security needs to be an ongoing process. It needs to be a state of mind, right? Somebody on the team, or ideally everybody on the team has the ability to look at the changes that they’re making in the Salesforce environment from a security perspective, right, because it is very hard to scale security people with developers. There’s more developers. The ratios is, I think, 500 to one. So there is one security professional to every 500 developers. So there is no way that you could have a good ratio, but enabling the developers so they can build securely from the ground up is something that is a very good practice. And security, like I said, is a process. It’s not a panacea.
Meighan Brodkey:
And question, just as a side joke, is it kind of like they fly through the Da Vinci Code as a Salesforce hacker, just like when everyone else is hacking the planet? Or is it a little bit different? Hackers reference, so sorry if anyone else has seen that movie.
Paul McCollum:
You said Da Vinci Code.
Meighan Brodkey:
The da Vinci pictures, sorry.
Paul McCollum:
The da Vinci virus. Yeah.
Meighan Brodkey:
Yeah. The da Vinci virus.
Paul McCollum:
Shannon had a comment.
Meighan:
Yeah.
Paul McCollum:
[crosstalk 01:12:55].
Shannon Smith:
Your Mac crashes twice a year, right, when you actually want to say something. I had a hard crash, but yeah.
Shannon Smith:
I thought that the AppExchange question was really interesting.
Shannon Smith:
First of all, that’s kind of what brought Waqas and I to this ecosystem together, doing pen tests for AppExchange, people that failed the AppExchange review.
Shannon Smith:
So we’ve been dealing with this for almost 10 years now, but we’ve been doing some research over the last couple of years, Waqas and his team. And we’re doing another run right now. We don’t have the exact stats, but many of the apps on AppExchange have vulnerabilities in them through third-party software libraries. Obviously, they passed a code test, so their code is good, which is great. But almost all the apps in AppExchange use third party software libraries like jQuery and all kinds of normal libraries that people use in app development.
Shannon Smith:
What the problem is those libraries become insecure out on the net, publicly posted CVEs globally known. But even if a company is really on top of their game and they’re like, “ooh, our app and that library is insecure”, but it takes them a while to update the app and get it back into the AppExchange, re-reviewed and re-posted. So that process can sometimes take months, right? And as a result, many of the apps in the AppExchange have vulnerabilities. And when you install them in your org, you inherit those vulnerabilities.
Shannon Smith:
So, you can find those by using an SCA or software composition analysis scanner. But yeah, I’ve talked to people all the time. They’re like, “oh, I don’t do any development of my org so I’m totally secure”. I’m like, “well, do you use apps?” “Oh yeah, we have seven apps.” I was like, “all right, well, you’re not secure then”. I’ve seen one app with one bad library and there is 1500 high risks in an org. So, you got to be cognizant of your apps, use good apps. Don’t use apps that haven’t been updated for years and think you’re safe.
Meighan Brodkey:
Nice. Agreed. And again, just be careful about the AppExchange listings you find via Google instead of via the AppExchange. That’s one way to tell what is bonkers’s charity review. As a tip, items that are found via the AppExchange search bar at the top, those ones have gone through the security review. If you find it via Google, most of them that have the little orange bar across the top have not gone through a security review. And that’s the way to tell the difference between the two. But you can still get on AppExchange without going through security review. That’s just that you’re not in the same search. You find them differently.
Shannon Smith:
Yeah. That’s a whole ‘nother category, those apps. No guarantee on those apps, but even the good apps that you all know and love, from the billion dollar companies that we all trust. They know when their app becomes insecure due to a library, but it still takes them weeks or months to get that updated, sometimes longer. So yeah, it’s a mixed bag, but the majority of the apps out there have some sort of, it’s the same thing with phones. It takes a lot of time to update phones and everyone buys a new phone every year. So the whole idea of mobile security is really an oxymoron because the update cycles for phones is so long that by the time a reported bug gets fixed and gets through all the carriers and all the devices, everyone’s buying new phones. So the majority of the time you have your phone, it’s insecure.
Meighan Brodkey:
Nice. And then, I like your comments, Daniel, that apps are native also aren’t a hundred percent assumed to be without vulnerabilities. So true, so true. We’ve gotten questions to slow down a bit. I think it would be great to check out that demo. What are you guys thinking?
Shannon Smith:
Yeah. Waqas, if I bring it up, will you take us through?
Waqas Nazir:
Sure, absolutely. Let’s do it.
Shannon Smith:
Right. Can I share my screen here? There we go. Alright. Okay. Let me stop my video. I’ll let you introduce it and then tell me when to hit play.
Waqas Nazir:
Sure. So, I touched upon this as we were starting about this exploit that we show. It exploits a common feature, which is called web-to-lead. You must be very familiar with it, so I’m not going to go into explaining what that is, but this is exploiting two issues with Salesforce. And the first exploit deals with input validation and the way data is rendered within Visualforce pages. And the second one deals with the way code is run on the Salesforce backend. So go ahead, Shannon, if you can play, I can go over the exploit.
Waqas Nazir:
So you have a basic Contact Us form on the website, where instead of sending good data or sending an exploit, which, when loads on the Salesforce side, we’re able to grab a user’s session remotely.
Waqas Nazir:
And once we do that, we can use this user’s session to then query Salesforce for data. And then you can do this via the Salesforce API, if it’s enabled or do it in the browser or whichever way you like it. We’re just going to use the API, because you can just write cURL statements in the API. So the first one is just select accounts. And we see that this user has access to some accounts, although there’s no test data. So there is no actual data that’s being used in this exploit. Then we look up who the user is that we’ve just compromised. An easy way to do that is just create the Chatter API. And we get the user’s email and some information about the user. You can also look at their photo if they have one set up. But given that we know it’s an internal user, it has some access to some data.
Waqas Nazir:
The next step for us is to identify how much data we can get to. One way of doing that is basically querying Salesforce for all different Apex controllers, Visualforce pages, things like that, which have been deployed within the org. That’s where you can find a lot of exploits. So we query the Salesforce API to get the list of all the available Visualforce pages. An easy way to tell that this user has the correct permissions to invoke this page just to get requests, and if Salesforce responds with a error or a 500 or 403, then we know that this particular page is not accessible. But if you get a 200 back, then means that the page is accessible. The first request was not “you got an error”. For the second one, we will get a response from the Salesforce API saying that “this page is accessible”.
Waqas Nazir:
So now we’ll take the session ID that we had compromised. After going through this exercise, we can identify what’s available, what’s not. We’ll basically plug this value into a browser and load the user that we’ve just compromised. And so this is a user we had compromised. It’s a standard platform user profile. The user has access, but it’s not all the access. So we come across this page, which allows the user to change their profiles. It doesn’t have some preset values in that pick list.
Waqas Nazir:
So what we will do is we’ll select a value from the pick list. That’s they as a partner user to switch the profile to, and the request will go through a proxy. And then we will change the parameters in the request. So basically what we’re trying to identify is if the actual code on the backend is doing any validation of these pick list values, or if it’s making sure that the user doesn’t escalate privileges and we’ll change the selected value from the pick list, from a partner user to a system administrator. And then we’ll send this request off to the Salesforce deployment.
Waqas Nazir:
So you’ve got the request, we’ll plug it into the payload that we had stopped in the proxy, and then we’ll send it off to Salesforce. And then we’ll look at the response that comes back from that controller. And we see that it was a success. So the request was processed. Now we have to go and basically refresh the page to actually make sure that it actually changed the profile or not.
Paul McCollum:
Waqas, can I pause you for a second and narrate and maybe catch people up? If you’re running a video, then.
Waqas Nazir:
Sure, absolutely.
Waqas Nazir:
No no, sure, go ahead.
Shannon Smith:
The video is done now. He basically got system admin. He owned the org with those two.
Paul McCollum:
Oh, okay. Sorry. Yeah. popped in right at the good part of the show. So, just talking through the tools that you were using and how this was going about, I think you used a wire sniffer to grab a session client from an active user or any active user. And then you used a couple other tools that were just looking for pages that were allowed to be accessed. Sorry, I’m just trying to put in a little plain user terms the anatomy of the attack.
Waqas Nazir:
Yeah. So, the first exploit basically is a cross-site scripting exploit, which basically injects malicious JavaScript into a Visualforce page that’s rendering a lead. And once that happens on that page, the user session ID is available as part of that page. And then we grabbed that. So, the JavaScript is designed to send that and send it to a server of the attacker, so once that happens, we went to this screen, which was basically a server listening rate for that exploit to work. And once it works, it basically gives you that session ID of that user. And once we’re able to get that, then we go down using basic cURL commands, which is to request stuff from the Salesforce API, where you can just construct SOQL statements and get data in JSON format.
Waqas Nazir:
And once we did that exercise, then we went into the browser and we used the browser and the web proxy, which basically allows you to meddle with requests before they’re sent to a server. So the proxy lives between a user’s browser and the Salesforce server, so you’re able to manipulate parameters. If the UI doesn’t allow you to change something, you can change it in the proxy. So you can change all sorts of parameters and find all kinds of bugs through that.
Waqas Nazir:
So those are the key tools used in the overall demonstration.
Paul McCollum:
Yeah, cURL is a very common command line-based web request tool. We use that for REST debugging. So, for anybody that this looks like hardcore hacking, these are really fairly basic tools, crafting custom URLs to get information that we don’t see in the browser, but it’s very, very much there and exposed in the interaction behind the scenes.
Shannon Smith:
Exactly. This is all open stuff. I mean, this is a canned hack. And obviously if a real hacker was attacking this org and this website, it would take them more than five minutes to figure this out, because they didn’t construct the maze like Waqas did, but it’s a real website. It’s a real Salesforce instance. These are two real vulnerabilities that we see all the time. And a real hacker might take whatever, hours, maybe a day or two, to figure this out, probably hours if they’re a good hacker at least. But this is definitely a very realistic scenario and it’s not super sophisticated from a hacker point of view.
Meighan Brodkey:
Sorry. Waqas I’m going to ask you the same question I asked you when I saw this demo the first time. So this is web-to-lead, if you’re using like an out-of-the-box lightning action, do you have the same type of vulnerabilities?
Waqas Nazir:
So it really depends on the controller that it’s calling.
Meighan Brodkey:
No controller. We’re talking out-of-the-box declarative action.
Waqas Nazir:
Yeah, then, there is no issues, because the object permissions get enforced. But if you have, what is it called?, a lightning web component action which you can have a controller, then you can have a similar scenario.
Meighan Brodkey:
So for when the rest of you are building, keep in mind, try to stick with that out-of-box, if you can, just another advantage, right?
Waqas Nazir:
Exactly.
Shannon Smith:
Salesforce out-of-the-box is, I mean, they do their security hardcore, internally.
Waqas Nazir:
Yeah. And I mean, these things are available and if done right, they can be very secure. But out-of-the-box features, of course, have built-in security so you can have a better bang for your buck. But the fact that not every use case is satisfied by out-of-the-box functionality, is where a lot of the security bugs come up as well.
Paul McCollum:
So can I ask? The cross-site scripting attack that launched this whole thing, how hard is that to implement or how worried should we be that, that’s possibly going on? To me that… I hate the cross-site scripting protections because I’m a heavy JavaScript iframer and love to be able to pull resources in. So I am annoyed constantly by XSS security preventions. So knowing that, that all is out there now, what does it take to get a cross-site scripting attack onto a webpage? Or what’s that initial foot in the door?
Meighan Brodkey:
And really quick, to add to your question, what about the options to do that for all from this domain, versus all from this domain plus subdomains, versus this domain plus all the sites that I add for cross-site scripting setting? We’re doing a lot of work in communities. I know we’ve got the cross-site script saying, protection sites and that where we can add to the list, and we can have just those sites we add to the list at it. Want to add that into the question as, for that side of the danger too. And where does that fall in the security risk area?
Waqas Nazir:
Yeah, so the cross-site scripting protections that are part of the platform, basically protect you against an actual exploit. So it doesn’t directly takeaway all cross-site scripting, it protects you from like somebody being able to execute JavaScript, for example, right? So the first step that we did was executing JavaScript, right? So if you take that ability away, it’s not that somebody cannot exploit you, they won’t be able to execute JavaScript. They can still build HTML elements, right? And then have the user interact with those elements, right? So they’re still able to inject things, but then you’re getting protection from like JavaScript execution. So those controls basically make it hard for somebody to exploit cross-site scripting. It doesn’t take away the fact that you have a process scripting vulnerability within an app. So I think from a user’s perspective, you must… It’s ideal to fix those bugs instead of relying on the taking one ability of it from an attacker to think that it’s more protected. But those things are actually very effective in stopping some attacks, some common attacks.
Waqas Nazir:
So I think as an admin, it’s important to enable those because they protect your users sessions. The other thing interesting with cross-site scripting is that when something happens on a site that your customer owns or you own, there’s a trust associated with it, right? So for example, if I am on Amazon site, I trust Amazon. So all the web components or the web elements, they are presenting with me, I trust them, right? So there’s a trust associated with the data that’s coming from Amazon. So the most attacks exploit that trust, right? So even if they are not able to execute JavaScript, or even if they’re not able to directly compromise a session, they would exploit the trust that a user has. And that may be just like redirecting them to a malicious site or redirecting them to or making them download something that they shouldn’t be able to download.
Waqas Nazir:
So the exploit, while there’s an element of it, is to compromise user sessions. But if you even have that covered and there’s little that can be done to compromise a user session, the trust can still be exploited, right? Even if you’re able to just put a statement there that this thing is on sale or something like there could be financial implications or where we’re yet to say something has happened, right? Similar scenario with famous people’s Twitter getting hacked, right? The content itself has a lot of value, so even if somebody can hijack that content, you still want to be careful with that attack vector. So it’s always best to fix the bug, than rely on protections which come with the platform, which are great, and I think should be enabled, but it’s kind of like whack-a-mole right? That, we’ll do this and if this doesn’t, this protects us, that’s fine. But if a new expert comes up, then we’ll try to mitigate this. But if you fix the issue at the root, then nobody will be able to exploit cross-site scripting in your own environment.
Paul McCollum:
So from a newbie, a little bit of a newbie, perspective, isn’t creating a cross-site scripting attack, have to do with being able to have a pull of malicious code from within the webpage. Don’t you have to have compromised through a banner ad or something to get your code, the malicious code onto that page, or call malicious code that’s not necessarily stored on that original domain from some other domain. So you’re taking… The original way that cross-site scripting happen, I believe, was banner providers would be compromised. And then whenever a website pulled in that banner comp or ad banner content, there would be additional content, and that was what would compromise the user and do things within the browser and within the session. So isn’t that first hack, getting foreign code embedded into the website somehow?
Waqas Nazir:
Yeah. So in Salesforce terms it’s basically writing to an object, right?, with that malicious data, which is in the lead object field. And then when that data gets rendered within Salesforce, vulnerability happens and that’s happening on a visual force page, which doesn’t encode that input properly, and that’s where it gets rendered and executed.
Paul McCollum:
Okay.
Waqas Nazir:
Does that make sense?
Paul McCollum:
Yes. Thank you very much.
Shannon Smith:
I have a question Waqas, on the cross-site scripting injection bug, if you will, or vulnerability, how would people detect that? Is it, I mean, can you get that from a source code analysis or is a config check, health check going to get that, or do you really need to do a runtime test to find that bug?
Waqas Nazir:
So you can find that bug both by doing some static code analysis and also by doing a runtime testing, right? Runtime is basically executing the way that we executed this exploit, right? Looking at all the inputs to a given Salesforce environment and sending crafted payloads, and then seeing the response and rendering that within that given visualforce page or component, and then seeing if our input was executed. So that’s actually one of the most effective ways of finding this bug, is to actually simulating the same thing that we did with this attack. And that’s how somebody would go in and identify a cross-site scripting bug. The other way is looking at the code and making sure that all the input that is coming from the user side, which is user controlled input, is properly rendered within the different HTML elements in Salesforce. And Salesforce provides native encoding capabilities to protect against those. So making sure that you call those when you’re rendering that data will protect you against things like that.
Meighan Brodkey:
And I know we’re kind of… we’ve been chatting for a while. I just kind of want to open it up to everybody. If you guys have questions, take yourselves off of mute. Opening up the floor so you guys don’t have to use the chat to ask questions. What else you guys got?
Meighan Brodkey:
So quiet.
Ryan Smith:
I don’t have a question, but I guess I have a comment. Shannon and I met with a CISO last week for coffee, and she’s on the board of this company in Chicago. And the CEO asked her, was asking her, about security and what he can do to… in his company. And she said, “The first thing is security’s a team sport”, she said, it’s not just for CISO, one person, she’s like, “Every person you hire empower them to be involved in the security process”. And I thought that was a really powerful thing, that it wasn’t just, it doesn’t just follow one person. It’s really, she said, “It’s the responsibility of everyone in the company”. And she encouraged everyone to be proactive in it. So I thought that was good advice.
Meighan Brodkey:
I like that. That security is a team sport, that’s a great way to put it. Just another thing about security, it’s okay to not know what you don’t know, and to admit it. I have to say probably one of the worst things to do when it comes to security is to say, well, I don’t know how to do it, so I’ll just open it up and then hide it. Probably the scariest org I ever walked into was a community, so we’re already walking into the – exposing our data to the outside world. And that community, they weren’t sure how to work the sharing and they couldn’t get it to work the way they wanted. So they gave everybody a view all and modify all, then decided to hide the data with components. And they were using components for a lot of it, but they were still using the out-of-the-box search.
Meighan Brodkey:
And you can still hack a URL in the community just like you can in Salesforce by dropping an ID at the end or switching the last character up. So people had view all and modify all, and if they could figure out what the URL was to the standard page, they could then go in and modify anything they wanted because nobody had decided to then, go through it and at least make all the fields read-only. So it was probably one of the most terrifying things I had ever seen in my life was, rather than say this isn’t what I know how to do, somebody deciding to wing it by opening everything up and they go in a heavy dev route, but still sticking with a bunch of out-of-the-box stuff. Terrified. I don’t know if anybody else has seen some security horror stories. It’s true what they say about making a career out of cleanup work for security.
Shannon Smith:
Yeah. Just the state of security I’ve seen in general for last 20 years, professionally, I’ve almost… It’s over 90% of the time a security pen test results in total compromise, whether it’s a network or an application. And I thought Salesforce would be different, but Salesforce has been no different. The general state of security across the world is not as good as it needs to be. And really it’s because we’ve relied on these senior IT people and security people to take care of it for us.
Shannon Smith:
And it’s back to what Ryan said, which is, at the end of the day, we’re all responsible for our own security and our own safety. And we all have to take some level of responsibility and knowledge, and learning just a little goes a very long way. So it’s amazing, if a good hacker comes at you, the majority of the time they’re going to have full access to everything. But little steps that everyone can take can make a big difference. It’s like that quote from Gandalf where it’s the little people that make the big differences and I really think that’s true.
Meighan Brodkey:
I am going to say one of my favorite quotes when it comes to Salesforce and the super powers that they give us, and I’ll just share it here on my screen. With great power comes great responsibility. Salesforce gives us so many options and so much power with the platform. But with that power, we have to be careful. And just because they give us the ability to do so much more and open things up for security, it’s for the instances in which we have to, it’s not for making it super, super open all the time. Like Waqas said, key thing is, permissions to a minimum.
Meighan Brodkey:
And one of my favorite things to do, is make sure to have people that are in a QA that know security, and know how to test for security, or at least help get some security scripts together. So they know how to do some negative tests and positive tests against security. And then also making sure that your devs are following sharing, and they’re only not sharing when they absolutely have to. Not following sharing when they absolutely have to. Some of those basic items, but yeah.
Paul McCollum:
So I would probably, Megan, to add on to that, promote that, with the growth of the ecosystem and the influx and massive growth of development resources, that the maturity path for a developer should be up to architect and then to security architect. Because people are entering the coding-work ecosystem and coding production code, before they understand the security layers that they are passing and working with. The stuff…I’ve been doing this since internet was command line and you could only see stuff on BBS. So I see where it’s grown up and I can see the layer changes and some of the stuff, a lot, but not all of what Waqas did during the hack makes sense to me. And I’ve been doing this for 40 years. People that are getting into it in two or three years are not going to have this perspective.
Paul McCollum:
And that’s what architects do is try and build workable solutions and tell more junior people how to assemble good high functioning solutions. And then our… As architects, we should be mature enough to know when, where, and how to secure them, because we’re aware of the depth of penetration of all the things we’re assembling happens at the security layer. And that’s just an aspect of architecture, but maybe kind of the pinnacle of architecture. And we need to continue to pitch that a key component of architecture and Salesforce solution and techARC, is security architecture, because people aren’t doing it and they just don’t have the experience to know that all of this stuff is going on. And it’s a side effect of the growth and being in a great ecosystem, that’s evolving so fast as we’re creating attack surfaces faster than we can understand them or learn that they’re there.
Meighan Brodkey:
All right. So I don’t know if you meant to say that a path for a developer should be to go to architect and then security architect. But I mean, if you say, I don’t know if I agree with that. I mean…
Paul McCollum:
Well, it doesn’t have to be everybody’s path, but you really, you have to-
Meighan Brodkey:
It’s totally different career paths. Developer is a developer path. Architect is an architect path.
Speaker 2:
Has anyone noticed that basically we’re getting a lot of really new people in the system, because developing pays a lot more than… And they don’t necessarily understand what it takes to be a good admin.
Meighan Brodkey:
Yes. And I would say that knowing administrative skills, so I could have sworn that back in the day, you had to get admin to get the old developer sure, before it was pd1, dev 400 and then dev 405. But now it seems like pd1, there’s no requirement to get the admin cert first. I think it’s vital to understand the platform as an admin, before becoming a developer. Not, you have to be at admin, but to at least understand the declarative side and the basic what is security for declarative?, what is the basic cloud for?, what is out there that we could do with clicks before code? And just choosing to become a developer because that’s where there’s a need or that’s where the money’s at, adds so many new devs that are out there that just don’t know Salesforce. I’m not saying that not all new devs know Salesforce. There’s just a bunch of them that don’t and it just seems to me they’re having to write super, super detailed tickets or a lot of turnover sometimes, I would say every time, but sometimes.
Paul McCollum:
So with these fifth gen, ninth and 10th generation languages, you’re coding so far from where the actual application happens. People don’t… The code abstracts it. We go in and create an object in a web interface and that sets up or enables a REST endpoint to talk to that object. Most admins aren’t going to know that. There’s no class really that tells you, Hey, by the way, you’re opening this up for public consumption for anybody to come in and poke at, if they’ve got a credential. And they can do it with API access, if you haven’t secured that. But even understanding all of the implications of the stuff you do is a rare talent, and they’re trying to make this stuff easier with flows. You don’t have to do memory management anymore. You used to have to maintain the size and allocation of variables as you were going through loops. They took that work out of it.
Meighan Brodkey:
Remember that count that you used to have to do?
Paul McCollum:
Yeah, with malick and garbage collection and stuff like that. So by taking these away, you remove the opportunity to learn how the system is actually working behind the scenes, and so that knowledge, just isn’t fostered anymore.
Shannon Smith:
Yep. And I just want to add-
Meighan Brodkey:
Everyone fully needs that though. As long as you know what you need to do, I don’t think you fully need to understand the back end.
Shannon Smith:
You made a really good point, Meighan. I mean, you don’t have to be an admin for five years before you can develop or whatever, but you need to understand what’s going on, on the operation side. I mean, the whole idea of DevOps and DevSecOps, it’s the final realization that we’ve been in this web world for over 10 years now, and people are developing stuff that immediately gets put on the network, right?, into operation. And this whole idea that you can just be a developer or you can just be an admin and no one needs to worry about security, is just not realistic for all the reasons that Paul just mentioned. This used to be different, right? People would develop stuff, and then it would get tested, and then it would get security tested, and then it would get put out on a network.
Shannon Smith:
And the network people were in charge of the security of the application running on the network. But now stuff gets produced, and a day later it’s being popped up into Salesforce or on the web and it’s live, right? And so this whole idea of, “We’ll test security once a year and it will be fine”, isn’t realistic. I mean, new security vulnerabilities, even external third party libraries, become exposed every day whether you’ve made any changes or not, right? So this is a continuous process. We’re all involved in to plan it, it’s the team together here, whether we like it or not. And code is not separate from admin. They’re all one and the same. Now you have to have at least an understanding of how these things are being deployed, if you’re going to develop for them.
Meighan Brodkey:
Yeah. And we do have another question about, what are some of the best ways to use the security center? So Waqas, I don’t know if you have any recommendations about the security center. I know that it’s great to do a scan, but it’s also pretty high level. Other feedback? I think Waqas might have stepped away, he’s gone to a picture and he’s on mute.
Waqas Nazir:
Sorry, yeah, what’s the question again? I’m sorry.
Meighan Brodkey:
Any tips on using the security center or recommendations to get the most value out of it? Well, the security scan.
Waqas Nazir:
Are you talking about force.com scanner?
Meighan Brodkey:
So let me scroll back up on the question. Sorry, it just went down. The SF security center.
Waqas Nazir:
Okay. This is basically, kind of, what you were talking about earlier around who can see what type of data. Also, it has the ability to look at authentication patterns and stuff like that. So I think this is definitely a good tool to utilize and detect some anomalies and things like that. But that’s around like identity management and access control. For that scenario, I think it’s very helpful.
Meighan Brodkey:
That answer your question?
Speaker 1:
It does, I was in a noisy area so I was I kept on mute all the time, sorry.
Meighan Brodkey:
No worries. All right. So I know we are at just after 11. I want to… This has been a great session everybody. Again, thank you so much Waqas for all of your feedback and all of your info. I think we all got a lot out of it today.
Waqas Nazir:
Thank you. Hopefully-
Meighan:
[inaudible 01:52:56] Sorry Waqas, go ahead.
Waqas Nazir:
No, go ahead Megan.
Meighan:
Oh, I was just saying, one last opportunity for questions everyone before we’re going to wrap up. Okay.
Ryan:
Great. Great job everyone. Megan, Paul, the whole team. Daniel, we appreciate you guys doing this and… It was a blast.
Shannon Smith:
Thank you.
Meighan:
Yeah. Thank you all for the time.
Paul McCollum:
Great information. Thank you.
Ryan Smith:
Please be well, everyone.
Shannon Smith:
Yeah. Well, if anybody ever wants to run a free security assessment, they can… Waqas has it ready on DigitSec for you, so give it a free trial there.
Meighan Brodkey:
Awesome. And then we’ll call Shannon. Ryan, do you guys want to stay on the line? Same with Daniel and Waqas?
Shannon Smith:
Sure, we’re happy to.
Meighan Brodkey:
Awesome. All right. Thank you everybody. You have a good o
