Confessions of a CISO: I Have Trust Issues

The speed of software development today is driven by fierce competition and the constant demand for innovation. Organizations are launching software faster than ever to keep up with the market and drive growth. This need for speed has led to several key trends:

• Greater Accountability Demanded of Developers: Developer productivity is no longer only measured by speed. Developers are now expected to play a bigger role in owning the quality and security of their own work.
• The Rise of Platform Engineering: Many development teams rely on platform engineering to build tools that enable developer autonomy. This allows them to self-serve and accelerate their building, testing, and shipping processes.
• The Cloud Operating Model: Cloud adoption has become an enabling standard, delivering reliable on-demand automation that can integrate the development process with enhanced proactive security and monitoring capabilities. This shift empowers developer freedom, but also demands increased accountability.
• Automation as a Requirement: Automation has a direct impact on the speed of delivery and thus has become an expected feature. Solutions that don’t offer automation – and in the near future, AI – will be replaced.

These trends introduce a critical dilemma: How do you balance speed vs. trust? While fast releases are essential to meet market and user demands, sacrificing trust for speed can lead to severe business repercussions. The “move fast and break things” mantra, while noble in spirit, can result in disastrous outcomes.

Trust is an organizational asset that is increasing in importance to enterprises, and is often referred to as the Trust Factor. Similar to brand reputation, Trust Factor is a key measure that reflects how much your customers, business partners, and employees trust your organization. A low Trust Factor can significantly compromise an organization’s ability to deliver on its mission.

Trust is crucial for both technical and business stakeholders, all the way up to the C-Suite (including CEOs, CIOs, CISOs, and CTOs). 93% of business executives polled in a 2024 PwC survey agreed that building and maintaining trust improves their bottom line.

What Happens When Trust is Compromised in Software?

Sacrificing trust in favor of speed can be very costly for businesses. In 2024, two separate incidents led to billions of USD in market losses:

Example 1: A content testing oversight at a cybersecurity technology company led to millions of crashed machines worldwide. This well-publicized incident caused market losses in the billions and ongoing lawsuits. The root cause was determined to be skipping a test to expedite the distribution of a data component. In the end, the company emerged stronger and more robust, but the event demonstrated that speed can hinder trust – and that it takes a lot of time, effort, and money to recover.

Example 2: Aggressive and untested updates from a leading audio technology company caused broken speakers and lost playlists. This disaster led to the CEO stepping down, layoffs, and a $500 million drop in the company’s value.

Both of these examples illustrate that everyone in the software supply chain is accountable for balancing speed and trust, not just the overall owners. Let’s take a deeper look at what this looks like in modern software development.

What are the Elements of a Trusted Software Factory?

Imagine software development as an industrial factory or manufacturing plant. To run this trusted “software factory” effectively, three key elements determine success: Ownership, Control, and Compliance.

We can think of these elements in terms of three key roles:

1. Ownership – Factory Worker (Developer, Data Scientist): They are the owners and creators of the actual software. In large organizations, there can be thousands of these workers who have individual ownership over specific features, microservices, or software releases, and are ultimately responsible to build, create, and ship software to customers with quality and security in mind.
2. Control – Factory Manager (DevOps, DevSecOps, MLOps): They instill and manage control points within the pipeline. They have oversight and control over the flow of software releases, providing resources, tools, and processes to help factory workers improve productivity, and ship products faster and safer.
3. Compliance – Regulator (CISO, CIO, MLOps, DevOps, GRC): These individuals are responsible for ensuring that the software factory builds reliable trusted solutions that deliver the expected outcomes, while complying with both internal company policies and external regulations. Their focus is the world of governance compliance, and business risk management, ensuring the factory operates according to the established rules and regulations for the organization, and that the product is safe to be used.

For this “software factory” system to achieve its goals, orchestration and collaboration are essential. Siloed teams, tool sprawl, and a lack of shared understanding of the software pipeline can significantly erode trust. Achieving end-to-end trust involves managing ownership, control, and compliance at every step of the software pipeline to prevent costly compromises.

Just like the manufacturing process behind the latest tech gadget, the journey of designing, optimizing, testing, and securing an end software application requires a streamlined, connected system of people, processes, and technologies. Any breakdown in the sequence leads to inefficiencies or faulty components that are expensive to fix later.

Issues related to trust are commonplace in the world of software development, and with the increased expectations around accountability, everyone from the business owners to each individual owner of each step in the “production line” plays a critical role.

What are the Main Challenges Scaling Software Development?

While the importance of Ownership, Control, and Compliance is clear, implementing them effectively at scale is challenging:

Managing Ownership at Scale

Organizations often manage hundreds, thousands, or even tens of thousands of software components that are used to deliver their application solutions or services. This makes it significantly challenging to track and manage each component’s impact on the overall application ecosystem. Current approaches involve building custom systems or integrating with external systems like ServiceNow, SAP, Jira, or Backstage to manage application identities. Some organizations even use naming conventions and properties within JFrog to capture ownership metadata.

However, these methods often involve manual effort, are prone to error, and may lack proper integration. They also become unwieldy when trying to correlate, measure and secure at an organization-wide ecosystem level. The ripple effect of a newly exposed vulnerability of an open-source software component becomes a huge undertaking, as everyone now needs to assess whether they’re using the impacted version, and if they use the vulnerable function in their solution. Therefore, a unified approach is essential for clarity, efficiency, and gaining visibility into each application.

Software Security and Quality Checks at Every Stage of the Production Line

A key challenge in software delivery is embedding security throughout the entire production lifecycle. Organizations can only scale effectively when they have consistent visibility and control at critical points. Security leaders should work to streamline the varied approaches to security checking and monitoring. When multiple paths to production exist, security risks increase exponentially. For this reason (along with other benefits like cost reduction, improved efficiency, and faster time to market), organizations should implement common controls along their production line.

Remember, this doesn’t mean each team should use their own tools, which is a legacy problem many organizations are trying to overcome. Teams should share common tools, similar to how IT and Security Operations teams use a centralized logging for event logs. By reducing complexity, we increase visibility which in turn allows us to reduce risk and increase our ability to trust our SDLC approach.

Putting Your Trust in Evidence to Achieve Compliance

Compliance demands tangible evidence – proof that policies have been followed, not just they exist. This proof needs to be linked together, forming a chain of evidence from the moment that the release is identified, all the way to release.

Given the complexity of the modern SDLC, which comprises an array of teams, tools, workflows, microservices, and development stages, capturing evidence in a reliable, scalable way is difficult for many organizations. Often, we hear of developers providing a screenshot image as a piece of evidence.

Ideally, most evidence should be generated automatically at each relevant stage. The various tools should trigger automatically, with outputs parsed to check for success or escalated to a human for further review. These tool outputs can typically be attached in XML or JSON format. Proving a release is “trusted” becomes much easier when you can demonstrate that each production release has passed all tests or received approved exceptions.

For this all to work, and to be acceptable to the downstream teams that can use this data (IT, Security, GRC, business owners, auditors, and regulators), all evidence must be:

• Clearly visible
• Artifact-bound (attesting to components and releases)
• Trusted (digitally signed for integrity)
• Open (allowing any type of evidence to be attached to an artifact for transparency and flexibility, like the JFrog Evidence Collection framework)

A Disconnect Between Deliverables and Dispersed Metadata

Fragmented metadata spread across disparate systems leads to operational burdens and increased risk. Organizations can face critical delays; for example, a customer told us that it used to take them an average of 5 days to determine who wrote a piece of software running in production. In the middle of an incident, such a delay is dangerous. If all the teams involved with building, securing and running these software solutions don’t have unified access to this critical data, then no one has visibility into the overall Trust Factor for the entire enterprise.

That loss of visibility and consistency (e.g., loss of metadata or releases) typically drives teams to individually develop or acquire their own solutions to derive this information. This in turn, creates a confused world of different result formats and outputs, ironically making it more complicated to create a unified view of the risk landscape, let alone maintain a unified (zero trust) access control with different permissions. A unified system of record is necessary to consolidate all application-related information.

What Best Practices Build Trust into Development Processes?

The key to mastering the balance between speed to market and building trust is to unify your software factory: bringing Ownership, Control, and Compliance together into one system of record.

Here are some actionable best practices to align your software factory:

Set a Foundation for Visibility and Governance

• Implement a unified, automated approach to track and manage application owners.
• Utilize a solution to collect evidence from anywhere in the SDLC, allowing you to prove adherence to any policy.
• Unify a system of record to connect metadata, building an audit trail you (and any auditor) can trust.
• Look at the data your teams generate, and see if there are overlaps to align data for new use cases.
• Identify how you measure your Trust Factor, and make it visible across the whole enterprise.

Integrate Security, Controls, and Policies

• Integrate both proactive and reactive controls to power security in your software from code to runtime.
• Develop and enforce a single path into production for all your software.

Encourage Collaboration and Alignment

• Ensure that all the teams involved in building, securing, and maintaining your software solutions have an understanding (and access) to the information generated across your software supply chain. This is not a case of “if we build it, they will come”, you have to be proactive in reaching out to them to engage them with this information.
• Create a unified group of collaborators who work together, not in silos.

The result of these efforts is the ability to truly trust your software and ML supply chain. You’ll be able to answer critical questions such as:

1. “Which teams are using a deprecated component?”
2. “Where is this open-source software package used?”
3. “Which team is responsible for introducing a vulnerability?”
4. “Which packages are vulnerable to this CVE?”
5. “What compliance policies are required to put application X in production?”
6. “Who overrode the blocking of this risk package?”
7. “How can I prove that a container was approved for external distribution?”

How Can I Get Started Shipping Trusted Applications Without Increasing Friction?

The ideal vision for modern software delivery is to have a framework of people, processes, and technologies working together. This way, you’ll manage risk with policies, and make business risk decisions based on hard evidence, at any scale.

This requires stakeholder alignment; everyone in the software factory needs to buy in. Software development is a team sport, and the team wins when everyone trusts each other and shares information. This trust begins with trusting the system of record around application development to improve communication.

We understand how critical trust is to your SDLC, and we hope this guidance provides a valuable starting point. Stay tuned for more insights, which we’ll be sharing at swampUP 2025.