Salesforce Security Blind Spots: Secure Observability & Logging

In the third in our Salesforce Security Blind Spots series, we had special guest Nikita Prokopev, Founder at Pharos, join Waqas to discuss more security blind spots around observability in Salesforce. Having observability for Salesforce is a must, but this observability can leave data out in the open. This is especially true with logging features, flows, and other applications.

Waqas kicks things off by stating the genius of Salesforce is its ability to be customized and be built-upon, from very small features to full-blown applications. But with each change and update, there is added complexity to the platform as well as heightened attack vectors from a security perspective. As a security company, DigitSec sees that many customers customize Salesforce heavily, with sometimes hundreds of changed being made each day.

Nikita adds that this is why observability and the ability to monitor your environment is so important. Especially when many customers not only want many changes, but want them fast. On top of this, there are also multiple departments handling these changes. With so many parts at play, securing your Org observability is a must.

The rest of the session focuses on:

• What makes up the observability framework
• Flow error email reporting and how it can contain sensitive data 
• Security and compliance implications related to this
• Tactics on how to close these observability blind spots.

These security experts dive into ways to practice secure observability with tips that include:

• Removing email logs out of the equation if possible
• Have tight security controls around who can see data from observability reports
• Redirect emails back into the Org so they never leave Salesforce

Please watch the full session above to get more insights, specific examples, and what you can do to mitigate observability risk.

Watch our last session in the series with Salesforce Distinguished Security Technical Architect, Rachel Beard and you can see our first session with Andy Ognenoff of Accenture here.

Full transcript below

Andy Montoya (00:10):

Hello everyone, and thank you so much for joining us today. My name is Andy from DigitSec, and we’re so excited to have our third in our Salesforce Security Blind Spot series. Today we’ll be focusing on secure observability, logging and other security related issues. And without further ado, I’d like to introduce Waqas and Nikita who will introduce themselves and then get things kicked off for us.

Waqas Nazir (00:36):
Hello. Good morning, everyone. Thanks Andy. My name is Waqas Nazir. I’m the founder and CEO of DigitSec. I’ve been engaged with Salesforce, you know, a little over 10 years, and have been engaged with security specifically around 20 years. I’m very excited to bring you a very important topic around observability and logging with our partner. Pharos, and turn it over to Nikita to introduce himself.

Nikita Prokopev (01:12):
Thanks Waqas and Andy. Hi everyone. Thanks for joining us today. I’ve been in the ecosystem since — in the Salesforce ecosystem — since about 2006. Started off actually working at Salesforce as an engineer. So that’s where my security background comes from. I’ve been on security teams building platforms, security related features like XSS framework, c-cert protection, wrote some authentication routines and have been a consultant for, for quite a while. A developer on the platform. And I turned product engineer and a founder of Pharos.

Waqas Nazir (01:58):
Welcome, looking forward to this.

Nikita Prokopev (02:00):
Thank You

Waqas Nazir (02:00):
Welcome, Nikita. It’s great to have you. So our series basically explores various features of Salesforce and various challenges that teams face when developing for Salesforce. Of course, the genius of the platform is its ability to be customized; the ability to build small features to very large, even full-blown applications. And we see with each change there is the added complexity to the platform. There’s the added attack vector from a security perspective. So it’s interesting to see how best Salesforce could be implemented so that it’s secure, it does the job and it’s actually efficient.

What we’ve seen as a security company which analyzes changes within Salesforce is that Salesforce is customized heavily. We see volumes of upwards of hundreds of changes in a day in a given Salesforce environment. So naturally you can see why understanding what the inner workings of those changes are is important, right? And that’s where observability comes into play, and that’s where the ability to monitor your environment comes into play.

Nikita Prokopev (03:35):
That’s absolutely right. I mean the customizability of the platform is kind of what, what sells it for most people, because everyone’s business is so unique and you have to build on the platform to get anything out of it. You just can’t use it out of the box. A lot of people don’t have the expertise to develop properly on Salesforce.

And so you see a lot of consulting engagements. I’ve been in the business for quite some time and it’s this: a lot of times what we see is this revolving door of, of consultants and, internal folks coming in, adding more to the complexity, layering that on top and then, after a number of years, you’ve got all this baggage, you’ve got all this logic that that no one really fully grasps. And I think that’s one of the scenarios that’s so common in the ecosystem.

Waqas Nazir (04:37):
Yeah absolutely! And I think that interestingly is a challenge from a security perspective as well. You have people of varying capability as it comes to security, and they’re developing and making those changes. Some will be, maybe, very well-versed in security, so they’ll be more aware of the impact of a change from a security perspective; while others may not be. So it’s important to do things in a more systematic and consistent way because as you know, people adopting Salesforce also want change fast.

That’s one of the benefits of having a customizable platform, which you can make changes and deploy within clicks. They used to have this mascot with the “No Software” mascot, basically. Basically the notion was that you don’t need to write code, you can just make small changes and iterative changes. So that kind of exacerbates the problem. Where you’re making these small changes, but over time, with each change comes this added layer of… you know, unaware of these… the impact of these changes. And sometimes it’s very hard for somebody who’s managing an environment to keep on top of it.

Nikita Prokopev (06:07):
Yeah, absolutely. And a lot of times the owners of Salesforce, within an organization, will be business units. Whether it’s sales, marketing, or operations. Usually the big driver for changes in these implementations is the business. And business wants to be agile. They do not want to spend time doing other things. They just want their stuff done. So that’s what you end up with. You end up with a lot of rushed code, a lot of rushed implementations, [all] for the sake of meeting whatever business goal. So it is important to keep all the proper engineering discipline, which is, let’s face it, there’s so much you can do on the platform. You can do back-end, front-end, you can do all kinds of integrations.

There’s a lot of mechanisms these days in Salesforce that are mature and complex. And if you don’t build right, you end up with security holes. Because, let’s face it, security is not…, It’s number 20 on the list of a sales ops person. They don’t really care about it that much. But once you get to a certain level of maturity in your organization, you have to now start paying attention to it. And you have to really start embracing that, the engineering principles, the process of DevOps, and all these great things that we know from traditional disciplines.

Waqas Nazir (07:44):
The security department within an enterprise, for them, security is number one. For DevOps folks, it’s like deploying fast. For sales ops, it’s getting the sales changes in. So finding that right balance by employing those engineering principles is important. And oftentimes, when it comes to Salesforce, what I’ve seen is that people do not understand the basic principles of what they have to implement from a security perspective. 

And I think we can talk about specifically from an observability standpoint, is this concept of the triple A in security. That’s the Authentication, Authorization and Auditing. And the third part, which is Auditing, revolves around observability. The ability to look at changes and things like that. That is actually all the customer’s responsibility. It’s not Salesforce’s. So all the changes that you’re pulling in, putting in place, YOU are responsible to observe those changes and the way those changes interact with your data. 

So it’s important to understand that, and I think that sets the level ground for us to delve into the topic today, which is observability and Nikita, that’s your wheelhouse. So I’ll turn it over to you to go over what you’re going to cover today.

Nikita Prokopev (09:03):
Absolutely. Thanks, Waqas. Yeah. So we’ll first introduce the concept here a little bit. We’ll try to cover a few topics around Observability. There’s a lot to talk about. As Waqas mentioned, it is the responsibility of the customer. Because Salesforce, let’s face it, does not give you much, in that way. And so a lot of times you have to, as an owner or engineer or architect, have to implement this yourself if you want that visibility. So for the most part, Observability is kind of an emerging concept in the Salesforce ecosystem for those reasons. There’s no coherent vision of it out of the box.

But there are a lot of settings that will give you some aspects of it. And, of course you can roll your own stuff. You can actually facilitate a functional observability framework. I’ve seen this many times in our projects. I’ve had to build this many times myself. So let’s introduce some terminology here. So what do we mean by Observability? Well, this actually comes from controlled theory, and it’s a fancy word for something that allows us a glimpse of the internal workings of a system, whether it’s from a debugging perspective, profiling perspective, air handling perspective, even security perspective. 

All of that kind of that notion is the Observability. And, so here on the slide, we have what we call the Pillars of Observability. Obviously you’ve got logs, you’ve got monitoring, you’ve got some metrics and visualizations around those notifications. There’s also another pillar that not here, but is commonly mentioned in this context, which is tracing.

So ability to get a full trace of a transaction and then analyze it. So all of that is sort of part of the Observability framework that we’re talking about. We’re going to focus on Logging and logs in general, because that’s kind of your lowest level building block of all these observability frameworks. And we’ll also get into some notifications a little bit. I don’t think we’re going to have too much time to talk about the intricacies of the implementation from development standpoint, but, we’ll focus on, the aspects of implementation from a security perspective. 

So, why don’t we start with something that I think most people will know. That’s something that comes with the platform. It’s been part of it from day one, as far as I can remember, at least, which is a long time. That’s Apex Exception emails. So, Waqas, you know, about these things?

Waqas Nazir (12:10):
Yes these are those detailed trace emails that you have to decipher and, they’re pretty interesting. So, anybody who’s done anything on the platform, I’m sure everybody’s seen these.

Nikita Prokopev (12:24):
Yeah, absolutely. And they typically get ignored because at some point, once you got, once you have a certain level of customization in the org, they just, the volume gets up there. Like, if you’re monitoring that inbox where all these stack traces come into, at some point you just give up. You just set up a filter in your Outlook or Gmail or whatever, and they just go away and you never see them again. 

But actually as annoying as they are, they’re quite useful, and they will tell you about unhandled errors in your code. You’ll get notified if there’s a governor limit breach, which is very important for any Salesforce architecture to know about these things. Flow errors come through email, email-to-case failures, web-to-lead… all of these things- the stuff that’s actually breaking in prod live. So they’re quite useful.

Waqas Nazir (13:27):
And I think it is kind of interesting that the approach to addressing this is where the challenge lies. It’s not the information that they’re sending, or it’s not the fact that they’re actually alerting you. You’ll say, that actually is a good feature of the platform, right? And how you can scale that and basically address it.

Nikita Prokopev (13:55):
Well, one of the things to look out for…We’ll, we’ll get into that, but yeah, you should definitely embrace these emails and make use of them. There are a couple ways to do that. You might think, “well, this is just error reporting,” right? But one thing you have got to keep in mind is actually “What is the content of these?” 

For things like Apex, there’s really not much, you know: you’ve got your stack trace, you’ve got basic information about the error. But one area that I think most people should pay attention to is a Flow Error. reporting. Flow is now a huge part of the platform. It’s given so many admins the superpowers [that] they can essentially not even have to worry about learning to code, because you can do all these things in Flows.

And Salesforce is heavily pushing that technology. They not only have Flows, now you’ve got Subflows. You can call Flows from Flows. There’s different types of Flows. You’ve got Out-of-Trigger flows, you’ve got the Screen flow. You can call these Flows from your LWC components now. So they give you all these tools. There’s Flow Orchestrator, which you can orchestrate multiple flows into a larger flow. There’s a lot you could do with these without writing a single line of code. Oh, and I think the recently they’ve introduced a callout, a beta feature out, which is you can do callouts from flows. You don’t even need invokable Apex for that. So it’s really a powerful tool and a lot of admins are embracing this and learning them.

There’s a lot of content about this stuff. But I’ll, I’ll give you an example here. So let’s say you have a Screen Flow. What is the Screen Flow? Screen Flow is basically a form that you build. They can be quite complex. You can have multiple steps in the screen flow, kind of like a wizard. So you capture some data, right? And then you either save it to a database or perform some actions or maybe a combination of all of the above. And so let’s say you have a screen flow that captures a wire transfer. We’ve actually seen this in some of our clients that do this, financial institutions. So let’s say I’m an internal person processing a wire transfer. I get the information from the customer over the phone or however and I enter this into this wire transfer form.

It doesn’t even have to be a wire transfer. It could be something less sensitive like a credit card application. So I’m talking on the phone with someone, I’m capturing their information to apply for a credit card, or even something as benign as a lead form. Well, guess what happens when the Flow breaks? When the Flow breaks, you get notified, just as we were saying, and you get messages about this in the email. Now, I don’t think that few admins know how this is reported in detail. But, by default, the last person to modify the flow gets the email. You can also set it up to go to the Apex Exception email alias. 

You can do a combination of things, but by default it goes to a single admin. Now the Apex Exception Email Alias is a very important part of the setup that you should absolutely pay attention to. This allows you to configure a recipient list of either users or you can add arbitrary emails that can receive these system errors. So with the Flow that I was talking about, the Screen Flow, when it breaks…

Well, you have a whole bunch of data filled in, and then let’s say it breaks at a certain step while Salesforce is going to give you the full dump, which is all of the stack trace. It will also give you data that you entered. All of that will go to an email. In the example of a wire transfer, that could include bank account information, in the case of a credit card application, it can be social security number that you’ve entered. Other things like even for a lead form, things like emails and phones and that kind of thing that could be considered PII (Personally Identifiable Information).

So all of that stuff is now flowing into your email. It’s not, it’s not the greatest thing. So now you’ve basically got a wire template sitting out there in your Exchange servers or in a cloud email system or stuff that you really do not want to get there. Because if you have a compromise on that side, then you know, all of a sudden people have access to more than they should. You can actually leak out the entire wire form. Like if it breaks at the very end, it’ll send you all the data that it gathered up to that point. So, you know, this is a very possible scenario, and it happens all the time.

Waqas Nazir (19:22):
That’s pretty big problem, right? From a security perspective, because not only do you have sensitive data, you’re actually causing a data exfiltration attack. So in the case of an email being compromised, you have sensitive data in that email, so naturally that data is actually compromised now. So it’s a pretty big concern when it comes from a security standpoint. The other thing is, from a compliance standpoint, like if you have PII sending in emails with GDPR and what have you, where you actually have to remove people’s data… How are you going to find that data to remove? It’s going to be a very difficult challenge to go through… Next to impossible, right?

To go and find all that data set to remove, which was just parked from exception. And what’s also interesting is what you said around callouts, right? Which impacts flows and which flows now enable. What’s interesting is if the callout fails, the authentication data is in that exception email as well. So not only do you have access to PII, you have access to other systems through this. So it’s [a] really interesting attack vector, [from a] security perspective.

Nikita Prokopev (20:33):
And you wouldn’t think that if you were building a system like that, like Salesforce, “Hey, let’s just send all this stuff through email”, that seems a little crazy. But I, I think due to the nature of how Salesforce evolved as a platform and how the maturity kind of grew… that email mechanism stuck around way longer than it should have. And everyone’s sort of used to it. They almost don’t notice it. Like I said, most people just ignore these emails, but you really shouldn’t. Pay attention to this! It’s something that seems minor, but the data that you can leak, and can be very dangerous.

Waqas Nazir (21:20):
So let’s shift gears a little bit. Look at how we can address this issue. Get into a little bit of logging and logging frameworks. So, given this challenge with the way the system is designed and what are some of the approaches that can be taken to address this?

Nikita Prokopev (21:44):
So, great question. What we’ve seen a number of different approaches. One from burying your head in the sand and just hoping for the best. But of course, that’s not what we’re here to talk about. For starters, the one thing you should do is you should eliminate email out of the equation. Because it’s just not a great way to analyze anything. And, of course, you have the potential security problems. So you have to have tight security controls around who can see that data and who you’d want to control where this data lives in the first place and for how long, right? Those are kind of all the basic things that we expect from kind of an observability standpoint. Now, one way to do that is to pipe these emails into an external system. Something that’s secure that will give you that analysis capability.

Something like Splunk or Datadog. Those are tools that have been around for a while. They have good reputation. Maybe your organization has some kind of data warehouse system on its own that you can use that instead of storing all that stuff in your inbox. It requires a bit of integration. So it’s not the most straightforward thing to implement, but it’s doable. However, the easiest thing that you can do… And that’s something that most people don’t realize, is how easy it actually is, is to basically redirect all this email traffic back into your org. So it never leaves the premises of your instance. And you could build that today. You can utilize what’s called an email service.

So if you don’t know what that is, you think of it as like an email-to-case. For general purpose stuff. So it’s an Apex class that will take an email content and will perform some actions, run some logic. So what you can do is you can save it into a log record when that comes in. So that way you prevent the data from escaping and you’ve got tracking now internally in your org. Which opens up all of a sudden a whole bunch of things that you can do with this data.

Waqas Nazir (24:22):
I think that’s a very interesting way of keeping the logs alongside the operation of the platform. I think there’s a benefit around doing that in terms of the ability to troubleshoot and view more in depth and [perform] analysis and things like that. But that doesn’t address the problems around having sensitive data in the log. You still have that issue, right? So what are some of the techniques or the ways that we can address that. For example if you’re still capturing the entire stack trace, basically capturing the same data through an email service, but now in a log, right? Instead of being in an email?

Nikita Prokopev (25:10):
Yeah, totally. That’s a valid point. The main difference is that now, all of a sudden, you have control over these data points, right? So even at the very basic level in Salesforce, you’ve got a lot of security controls that you can leverage. You can decide who gets access to log objects, right? You just, turn that off at the profile level. Assign permission sets to only people that need that access, which is not very many. So most people would not see any of the logs. You can also set up targeted notifications. Because one of the benefits of the email is that even though most people ignore it, you still get notified and you still have that in your inbox. So for those people that monitor, it’s a valuable tool.

So you would want to set up some sort of notification strategy around these logs. But you can target it. You don’t have to send out the whole body of the stack trace. You could just send out the error message or you could send out… “Hey, something happened, take a look at this log.” Even something like that would be a valid kind of notification approach. The other thing that this approach gives you is, the control over the retention policy, right? You’re not going to have these logs sitting around for ages and ages. 

You can purge them after a year. You can archive them after the X amount of days or whatever it is you want to do. Whatever is appropriate for your company policy. You’ve got other benefits besides the security, as far as the sensitive information. That doesn’t eliminate all the problems.

You still have that at the end of the day. Since we’re talking about problems here, I’ll add one more to the list. Those are just the automated system emails that we’re turning to logs, right? That’s one side of things. But also what we’ve seen a lot of times is developers that are building on the platform, they will want to do their own logging and at their discretion what they put in those logs.

So I’d like to switch gears a little bit and talk about the logging frameworks.

Waqas Nazir (27:31):
Sure. I think that that makes sense. There’s just the platform generated events and then there’s like operational logs, right? Which are typically written as part of your custom code, the Apex and what have you. So there’s that element as well.

Nikita Prokopev (27:49):
We’ll, we’ll circle back to the system stuff. We’ll kind of tie it together with the custom logs. I think most people are familiar with logging frameworks. It’s a utility of some sort; usually written in Apex that will create custom log objects with error messages, debug data, or, you know, anything that you really want to put in there. It’s usually leveraged within Apex, but a lot of folks these days are using it for flows because those things get ginormous and you really want to know what the execution looks like. You want to log some data in between. So it’s a useful debugging, profiling tool even for flows. There are a bunch of open source ones out there like Arrow Flip, Nelo, LAGR, all of these.

But a lot of times people will build their own which is totally fine at the end of the day, it’s not that difficult. The problem doesn’t really come up until people start using it. If you ask any developer or an admin, how much information would you like to have in the log record? They’ll tell you, “I want everything, because I know if something goes wrong and I have to retrace the steps and try to figure out a problem, I want to have as much info as I can because that gives me all these possibilities to investigate.” It’s just common sense, right? It helps you resolve things faster.

Well, the number one reason people are building custom logs is for things like integrations. Because a lot of times you may not have an exception that pops up in your code or you may not have a systems failure, but you’ll have an integration failure and you’ll send a request to an external service and it comes back with a response and then something just doesn’t go right. Then you have to sit there and figure out, “Okay, well what did I send? What was my response pay… What request payload? What was the response payload? I have no idea.” 

And then you have to play this game with debug logs and try to recreate it and you may not be able to recreate it. So it’s best to log during the failure. And what you typically log is that whole payload, which, you know, as you can probably imagine, is a little dangerous,

Waqas Nazir (30:17):
Right! It’s like the, the integrations around the request and the response is typically all the headers, all the information, the form itself, the parameters that is being submitted, sometimes the entire JSON request and the response. So it’s basically logging the entire thing in case something goes bad and you have to go and review which is like really from a logging perspective, it’ll be a challenge to go and review, if you’re logging every single thing. But yeah, you’re right. Those are so hard to debug that sometimes they want to take that hit of logging everything in the rare event that it’s very hard to debug.

Nikita Prokopev (31:04):
It’s kind of kind of nuts actually, because you could have what’s basically the equivalent of a password to an external system just kind of sitting around in your Salesforce log objects. I’m guilty of that myself. We’ve had to build an integration with Jira Atlassian service and their API, if you’ve ever worked with them. They’re a bit notorious for making changes and just kind of letting you deal with it. So we’ve actually had to log off tokens just to see what the heck is going on there, right? And that was the only option because we had no other visibility. Now the problem with that is it may not be a problem as such as in a controlled test environment.

But the challenge is that people will forget to take it out. You put in a log, you look at some code and there’s console dot log or system dot debug statements littered everywhere because developers have put it in during DEV and then they forgot to take him out. Now it’s made its way into mainline and all the way, all of a sudden, now it’s in Production. So even if you don’t mean to cause any harm, it, you know, humans are humans at the end of the day. So we’ll forget things and we’ll let these things slip. So that’s what happens.

Waqas Nazir (32:28):
I mean, there’s definitely some ways of addressing this, right? And one of the approaches is to not just log everything, but log what’s relevant. Which is a more conscious effort. Like in the case where you actually absolutely need to log a token, it’s fine to do it in a test environment to make sure there’s nothing wrong with that. But to log a token in production would, raise some questions. What are your thoughts around addressing this? The issue with logging the sensitivity of data, for example.

Nikita Prokopev (33:19):
So I think the most important thing… The first thing you have to take care of is to centralize all logging through a single entry point. With the example that we talked about earlier where you’ve got the emails getting piped back into the org through this email service, use the logging framework to do that logging for you. So when an email comes in, you create the log through that logging framework. When a developer wants to log something or an admin wants to log something in the flow, make sure they’re using the same set of tools for that. And what that gives you is that one centralized entry point. And then from here, now you have control over the data that’s being pushed into your org.

You could start doing interesting things, like data sanitization. So that there are certain things that absolutely should not be in the logs. We all know what credit card numbers look like, social security numbers, even emails, right? You may not want these things to be sitting out there in the log data because maybe that’s considered PII and your company policy does not allow storage of these things. Auth tokens, same thing, right? Like what your Org integrates with, you know what those tokens look like. You can, come up with a dictionary of patterns that you absolutely need to get to make sure they keep out of the logs.

So what you can do is you can build a bunch of regexes, essentially, and because you’ve centralized, your logging through one set of tools, you can implement the sanitization logic at that level, at the very lowest level. So before you write to the log, you run it through that dictionary of blacklist of these regex patterns and make sure you sanitize it. And by sanitize, I mean, you could just mask it, right? Or do things like that. But you’d be surprised that how much damage you can prevent by doing this very simple thing. These regexes aren’t even that complicated. It just takes a little bit of fiddling and you’ll get it,

Waqas Nazir (35:37):
And I actually, you can also look at it from marking some fields as sensitive. There’s the Salesforce feature around marking certain data as higher classification, so you can always filter on those so you’re never touching that data, putting it in a log. So it’s more of an enterprise-wise practice.

Nikita Prokopev (36:04):
That’s a good point. You can also address this with a process… your SDLC process, your DevOps process, where you don’t just let people, people’s changes through without any kind of scrutiny, right? So you have code reviews and peer reviews which are great because you get another set of eyes on the change. So you have that extra control step before any code gets out there, right? So obviously it involves another human, it’s going to be error prone, but at least it gives you some degree of assurance that “Hey, someone looked at that code, Hey, why are you logging that Oauth token here? Can you just log a hash of it?” Or things like that, right? We’ll have to become part of the review criteria and you just reject the Pull Request if there’s crazy stuff going on.

Waqas Nazir (37:06):
And also you can build automation around logging sensitive data to log in your, in your mechanisms? So you can actually write rules around that and make sure that this type of data doesn’t end up in this function call or this login framework.

Nikita Prokopev (37:27):
Absolutely.

Waqas Nazir (37:29):
So that another way of addressing that. Another interesting challenge, and I know we are going a little bit over time… We have questions as well… Is around people changing the logic of the logs. So let’s say a company agrees on “these are the things we need to log”, but a developer decides then, “I can block some additional things which would be interesting for me if things don’t work out” or something like that, right? So, how can we prevent the manipulation of… or using the logs system as an attack vector itself. Because technically logging is another customization built on top of Salesforce.

Nikita Prokopev (38:15):
Exactly! Yeah, that’s a great point because, you’ve got a disgruntled developer that wants to get back at the company for whatever reason. You have to actually protect against these malicious internal actors, right? They have access to the code base. They can go and just push it to Prod without bypassing the whole DevOps pipeline if you don’t have that tightened up. So one of the things that I’ve seen, and I actually really do like this approach, and Salesforce to some extent are pushing it themselves. That’s using managed packages, right? This is not… We’re not talking about listing stuff on app exchange here. The packaging part of it is separate from the listing part of it, which is the app exchange.

That’s kind of your marketplace, but you can build managed packages without even going to the app exchange. You might ask, “Well, why bother with the, with all the hassle? I mean, they’re, they’re complicated. There’s restrictions and things.” But one of the things that the managed package gives you is this ability to hide metadata. What you could do is you can not only hide your code. 

If you have a bunch of Apex and you don’t market global, no one’s ever going to see it on the subscriber site. So once you install that package, you can go to look at that Apex class, you’re not going to see anything. You might just see a skeleton of the methods and such, but that doesn’t really tell you much. But the other cool thing is that it lets you completely hide your logging routines, your data sanitization routines.

So if you have consultants working for you, just tell them to use that log detail that you’ve now put in a managed package and they don’t know anything about it. They just use it and it conforms to all of your standards and policies. So it’s a really nice way to black box things. Then you give few people control over that package. 

At the end of the day, someone has to maintain it. So your internal folks are the ones managing it and the consultants are just using it, or maybe even within your internal team you have designated people that you trust and that have good reputation that have been with a company for a while or whatever the criteria, the standards you want to use. But they’re the ones that are overseeing the changes that go into that managed package.

Waqas Nazir (40:44):
Now we’ve seen this approach being implemented, you know, really effectively not only does it, you know, address this code audit change and the ability to control the utility, but also the ability to kind of segregate sensitive metadata, right? Totally protected custom settings and things like that. So instead of exposing those, we like unmanaged code, you can use package based development. 

Nikita Prokopev (41:12):
You could do like a connector kind of package where if you want to talk to an external system and you want to put those tokens in protected custom settings that’s a great way to do it. The subscriber will never see it, but you can still leverage that package interface to talk to that, you know, whatever that service is, and without exposing anything. So it’s a really cool way. It’s kind of like Salesforce is pushing it because they think it’s a good paradigm for development, which it is, I think it could work for some use cases kind of like that – microservices, architecture and, and spirit.

Waqas Nazir (41:55):
So I think that’s some really good material that you shared here Nikita around the approach to doing observability more effectively within Salesforce. We’re coming up on time and we have at least one question that I can see that has come in, so we can jump gears here and take some questions. We were hoping to cover more material, but as always, there’s so much to cover. So maybe, in the future time we can talk more about it. So the question here is “How can observability and security awareness in Salesforce work together to help secure a Salesforce environment?”

Nikita Prokopev (42:44):
Oh, that’s a really good question, actually.

Waqas Nazir (42:49):
You take a stab and then I can chime in.

Nikita Prokopev (42:52):
Sure. Yeah. So, I think that in terms of what observability can do for you from, from a security perspective, we’ve talked about logging things, so far in the context of debugging and air handling, but it’s also a great way to capture events. Any sort of event that may be happening. For example, I’m accessing certain parts of the system, let’s log that. Who accessed this? At what point? I’m trying to perform an action, right? Or even in the context of these integration connectors, right? I’m talking to an external service. Let’s log this event, right? Let’s start building that audit log of all of the things that touch sensitive data that involve PII, it might involve other things, whatever is important to the company.

And I think most people know the nature of the business, right? Most of the internal teams know the nature of the business, so they know what’s important. So using that same login till you create an event log and then you can start putting in these security events and later on you can then this is great for audit. This is great for figuring out attack vectors; later on you can start reviewing these. So this person’s accessing this data so once you start analyzing this data, I think you’ll see a lot of patterns start emerging. 

And it’s a really good way to get insight into where you might have a problem or where there has been a problem that you just don’t know about yet.

Waqas Nazir (44:41):
Yeah, and I think that’s a really good point around like not knowing, which is paramount for security, right? You cannot ascertain whether your system is operating securely without the knowledge and that cannot come with observing the system. So they actually go hand in hand if you look at it. And from a security perspective, there may be certain events you’re more attuned to like report exports or invalid login terms or things like that. So certain events you’ll be more attuned to, but they should fall in your overall approach to observing the platform. 

Of course the security sensitive actions should go through the security department or security ops or what have you, right? And then the more operational information would go to the admins and the developers and things like that.

Nikita Prokopev (45:34):
Totally. Yeah, you can even control these event logs, give them different set of policies around who can access them versus your other debug logs and stuff like that. So, if you have the cash for it, you should get event monitoring, that there are a lot of good security events that are Salesforce themselves will log. It is not free though, so if you can swing it, I think it’s useful, but if not, you can roll your own.

Waqas Nazir (46:02):
So we have another question. So it’s basically managing multiple profiles for integration is a pain, especially with the future sunset of profiles as we know it. What is your thought on assigning integration users to minimum access Salesforce and then using permission sets to enable permissions on the individual integration users? You take a stab at it.

Nikita Prokopev (46:30):
I’ll let you take that one because seems like that’s in your wheelhouse.

Waqas Nazir (46:33):
That is a good approach because what you’re doing is that you’re building from the least privileged model. And that’s what Salesforce is trying to promote. And I think one of the reasons why profiles are being sunset is not everybody needs that access, but if it’s assigned to a profile, then everybody who’s a part of that profile gets that access. So it’s being addressed by enabling specific permission sets. And you start with the least privilege and then you add what’s needed. And then there’s, of course, it becomes hard to manage over time. That’s why they have the concept of permission set groups. 

So you can actually have the same profile-ish type thing, but it’s more controlled. Because now you can group these permission sets together to create that one profile that you always need for a certain integration and then use that across the board. Actually it’s a better approach than creating an account and giving it a profile and then using that for integration and things like that, which has been done in the past. That’s a really good question.I know we’re up on time. Andy, do we have any more questions?

Andy Montoya (47:51):
Yes, just one more that we had here and I think after this when we can close things out. What were the specific notifications that can contain sensitive data?

Nikita Prokopev (48:03):
It’s mostly around flows. So that first example that I provided there was a screen flow failure. And the reason I picked on screen flows first and foremost is it’s… auto launch flows are guilty of that too… but with the screen flow, you’re directly passing user input into an email message. And depending on what step the failure occurs in the flow you’ll get a certain snapshot of data. But, if you filled out the form pretty much entirely and then you hit that last submit button and then something breaks, basically the whole form goes out to the email and that can contain anything really, it just depends on the nature of that screen flow. But the one that is notorious is Apex Exceptions. 

An Apex Stack Trace that’s not really going to give you, that’s not too sensitive. I mean, it does tell you the code stack trace, but what can you do with that information when you probably do something, but not as bad as like a bank account number leaking, stuff like that.

Waqas Nazir (49:20):
I think even from, as you were saying, an Apex standpoint, the stack trace can be interesting, especially if it’s generated from an auto launch flow. Cause if you’re reading stuff that’s sensitive, it’ll actually have that right as part of the flow that “Hey, you said get records. Oh yeah, and this is the record I got.”

Nikita Prokopev (49:46):
It will give you the data too. Yeah. It’s just it may not be user entered data, but it’s data that lives in your records. Yeah. Auto launch flow will absolutely expose that.

Waqas Nazir (49:56):
And I think it’s important to kind of emphasize on screen flows, especially because it’s easy to visualize, “Hey, somebody just entered this and it’s in somebody’s inbox now, right?” Yep. It didn’t even touch Salesforce. So you have no trace of this data going through your system and now sitting in the inbox. So I think it’s a very good visualizing of the problem

Nikita Prokopev (50:19):
And because these email notifications… because no one really manages them to the extent that they should, people forget who the recipients are in the first place, right? So it’s very easy to like, “Oh, just add a bunch of users, Oh, add this email, this consultant wants to see your error logs. Great, let’s add their email too.” You can get really carried away and if you don’t manage this thing closely, at some point you’re just not going to know who’s receiving these. Right. That’s a big problem!

Waqas Nazir (50:49):
Right.

Andy Montoya (50:52):
Well team, I think that’s it. The last comment we had is another great security blind spots webinar. So I think with that it’s perfect timing to close things out. Any last parting words, Waqas or Nikita, before we close things out?

Waqas Nazir (51:06):
Well thank you everybody for joining us. We appreciate your time and feel free to share your thoughts and insights around logging and security. We excited to hear more.

Nikita Prokopev (51:16):
Yeah, same. Thanks everyone. Really appreciate your attendance. Like Waqas said, if you have any questions we’ll be around to answer those. Happy to talk about this. We can talk about this stuff forever. We had a lot more content, but you know, again, we might have to do round two on this.

Andy Montoya (51:38):
Sounds good. Thank you. Well thank you guys so much.

Waqas Nazir (51:42):
Take care. Bye.

Nikita Prokopev (51:43):
Thanks.