[INS-206] Store Gitlab Project ID in secret location metadata #4601

mustansir14 · 2025-12-11T14:32:35Z

Description:

This PR adds the changes to include gitlab project details like project ID, project name and project owner name to the metadata of a chunk.

Achieving this wasn't straightforward as the chunks are generated by the git source and it calls an injected SourceMetadataFunc to create the metadata. The signature of this function obviously does not include the gitlab project ID.

The solution implemented here is to maintain a repoToProjectCache map that stores the project details for a repo. This cache is populated as we enumerate the repos and the callback function uses this cache to populate the project details.

The only concerning part of this solution is memory usage, so just to put in perspective how much this impacts that, I found a public organization with ~3000 projects and ran benchmarks for before and after making the changes. (I know this isn't anywhere close to the largest organizations we've come across, but this is the biggest one I could find that was public)

Results before making changes:

Total memory usage: 43234992 bytes (43.23 MBs)

Running tool: /usr/local/go/bin/go test -benchmem -run=^$ -bench ^BenchmarkMemoryUsage$ github.com/trufflesecurity/trufflehog/v3/pkg/sources/gitlab

2025-12-11T18:58:01+05:00	info-0	context	starting projects enumeration	{"list_options": {"pagination":"keyset","order_by":"id","sort":"asc"}, "all_available": false}
2025-12-11T18:59:01+05:00	info-0	context	Enumerated GitLab projects	{"count": 2930}
Memory used during benchmark:
Allocated (Alloc): 43234992 bytes
Total Allocated (TotalAlloc): 43234992 bytes
Memory obtained from system (Sys): 17629184 bytes
goos: linux
goarch: amd64
pkg: github.com/trufflesecurity/trufflehog/v3/pkg/sources/gitlab
cpu: 12th Gen Intel(R) Core(TM) i5-1235U
BenchmarkMemoryUsage-12    	       1	59642136351 ns/op	42916920 B/op	  246843 allocs/op
PASS
ok  	github.com/trufflesecurity/trufflehog/v3/pkg/sources/gitlab	60.146s

Results after this change:

Total memory usage: 44666216 bytes (44.66 MBs)

Running tool: /usr/local/go/bin/go test -benchmem -run=^$ -bench ^BenchmarkMemoryUsage$ github.com/trufflesecurity/trufflehog/v3/pkg/sources/gitlab

goos: linux
goarch: amd64
pkg: github.com/trufflesecurity/trufflehog/v3/pkg/sources/gitlab
cpu: 12th Gen Intel(R) Core(TM) i5-1235U
=== RUN   BenchmarkMemoryUsage
BenchmarkMemoryUsage
2025-12-11T18:38:02+05:00       info-0  context starting projects enumeration   {"list_options": {"pagination":"keyset","order_by":"id","sort":"asc"}, "all_available": false}
2025-12-11T18:39:09+05:00       info-0  context Enumerated GitLab projects      {"count": 2930}
Memory used during benchmark:
Allocated (Alloc): 44666216 bytes
Total Allocated (TotalAlloc): 44666216 bytes
Memory obtained from system (Sys): 21823488 bytes
BenchmarkMemoryUsage-12                1        66827846825 ns/op       44350440 B/op     261384 allocs/op
PASS
ok      github.com/trufflesecurity/trufflehog/v3/pkg/sources/gitlab     67.355s

It's important to note that this doesn't really perform a full scan (I tried doing that first but it would take hours), so I tweaked the code to expose the callback function and directly called that after enumerating the repos. This is the code used for benchmarking:

func BenchmarkMemoryUsage(b *testing.B) {
	// Enable memory allocation reporting
	b.ReportAllocs()

	// Run a specific function and measure memory usage
	var mStart, mEnd runtime.MemStats

	// Get memory stats before execution
	runtime.ReadMemStats(&mStart)

	connection := &sourcespb.GitLab{
		Endpoint: "https://gitlab.eclipse.org",
	}

	s := Source{}

	conn, err := anypb.New(connection)
	if err != nil {
		b.Fatal(err)
	}

	err = s.Init(context.Background(), "benchmark", 0, 0, false, conn, 100)
	if err != nil {
		b.Errorf("Source.Init() error = %v", err)
		return
	}

	feature.UseSimplifiedGitlabEnumeration.Store(true)

	// Run the benchmark loop
	b.ResetTimer()
	for i := 0; i < b.N; i++ {
		reporter := &sourcestest.TestReporter{}
		err = s.Enumerate(context.Background(), reporter)
		if err != nil {
			b.Errorf("Source.Chunks() error = %v", err)
			return
		}

		for _, unit := range reporter.Units {
			repository, _ := unit.SourceUnitID()
			_ = s.cfg.SourceMetadataFunc("test file", "test email", "test commit", "test timestamp", repository, "test local path", 1)
		}
	}

	// Get memory stats after execution
	runtime.ReadMemStats(&mEnd)

	// Calculate the memory usage
	allocDiff := mEnd.Alloc - mStart.Alloc
	totalAllocDiff := mEnd.TotalAlloc - mStart.TotalAlloc
	sysDiff := mEnd.Sys - mStart.Sys

	// Log the memory stats
	fmt.Printf("Memory used during benchmark:\n")
	fmt.Printf("Allocated (Alloc): %d bytes\n", allocDiff)
	fmt.Printf("Total Allocated (TotalAlloc): %d bytes\n", totalAllocDiff)
	fmt.Printf("Memory obtained from system (Sys): %d bytes\n", sysDiff)
}

Checklist:

Tests passing (make test-community)?
Lint passing (make lint this requires golangci-lint)?

…cation-metadata

amanfcp

LGTM! A few suggestions

amanfcp · 2025-12-15T08:59:59Z

pkg/sources/gitlab/gitlab.go

+	// Cache project metadata if available.
+	if gitUnit, ok := unit.(git.SourceUnit); ok && gitUnit.Metadata != nil {
+		if gitUnit.Metadata["project_id"] != "" {
+			project := &project{}
+			projectId, err := strconv.Atoi(gitUnit.Metadata["project_id"])
+			if err != nil {
+				ctx.Logger().Error(err, "could not convert project_id metadata to int", "project_id", gitUnit.Metadata["project_id"])
+			} else {
+				project.id = projectId
+			}
+			project.name = gitUnit.Metadata["project_name"]
+			project.owner = gitUnit.Metadata["project_owner"]
+			s.repoToProjCache.set(repoURL, project)
+		}
+
+	}


Would recommend early return flow here. It's kind of difficult to read 😅

Hmm, totally agree. I'll make the changes

amanfcp · 2025-12-15T09:00:40Z

pkg/sources/gitlab/gitlab.go

+	if proj.Owner != nil {
+		metadata["project_owner"] = proj.Owner.Email
+	}


We're missing the username check here as handled in gitlabProjectToCacheProject

Great catch, thanks

…suggestions

…data' of mustansir:mustansir14/trufflehog into INS-206-Store-GitLab-project-ID-in-secret-location-metadata

rosecodym

There is some scanning pipeline machinery that cannot yet support this, so this PR won't work as-is. I'm requesting changes until I (or maybe @mcastorina) has some time to go through it and figure out whether "won't work" actually means there will be problems, or it just won't work.

rosecodym

This looks pretty straightforward, but I have some notes and questions.

rosecodym · 2025-12-16T18:30:39Z

pkg/sources/gitlab/gitlab.go

+	}
+	// extract project path from repo URL
+	// https://gitlab.com/testermctestface/testy.git => testermctestface/testy
+	repoPath := strings.TrimSuffix(strings.Join(strings.Split(repoUrl, "/")[len(strings.Split(repoUrl, "/"))-2:], "/"), ".git")


Did you consider properly parsing this as a URL instead of doing a simple string split?

Relatedly, this looks like it will panic if the repo URL is sufficiently malformed. Even if string splitting is the best solution, we should avoid that.

Thanks for this! You're right, parsing it as a URL would make it much more reliable. I'll make the changes.

rosecodym · 2025-12-16T18:36:04Z

pkg/sources/gitlab/gitlab.go

 		}

+		// Clear the repo to project cache when done.
+		defer s.repoToProjCache.clear()


While I respect your concern for cache size, I don't believe clearing this cache is necessary. Source structures don't survive past a single scan, so clearing them at the end of the scan doesn't reclaim any memory that wouldn't be soon reclaimed anyway, and introduces the possibility of race conditions in the case that the git-parsing code sends work to separate goroutines.

repoToProjCache uses a mutex internally so there wouldn't be a possibility of race conditions, but yeah if a Source is always used for a single scan then freeing up the memory doesn't make sense. I'll remove that.

rosecodym · 2025-12-16T18:39:44Z

pkg/sources/gitlab/gitlab.go

+type project struct {
+	id    int
+	name  string
+	owner string
+}
+
+type repoToProjectCache struct {
+	sync.RWMutex
+
+	cache map[string]*project
+}
+
+func (r *repoToProjectCache) get(repo string) (*project, bool) {
+	r.RLock()
+	defer r.RUnlock()
+	proj, ok := r.cache[repo]
+	return proj, ok
+}
+
+func (r *repoToProjectCache) set(repo string, proj *project) {
+	r.Lock()
+	defer r.Unlock()
+
+	r.cache[repo] = proj
+}
+
+func (r *repoToProjectCache) del(repo string) {
+	r.Lock()
+	defer r.Unlock()
+
+	delete(r.cache, repo)
+}
+
+func (r *repoToProjectCache) clear() {
+	r.Lock()
+	defer r.Unlock()
+
+	clear(r.cache)


What do you think of moving this to a separate file (still within this package)?

rosecodym · 2025-12-16T18:50:08Z

pkg/sources/gitlab/gitlab.go

+		// ensure project details for specified repos are cached
+		// this is required to populate metadata during chunking
+		for _, repo := range repos {
+			s.ensureProjectInCache(ctx, repo)


It took me a few passes back and forth to figure out why sometimes you cache projects using ensureProjectInCache, and sometimes you directly cache the the value returned from gitlabProjectToCacheProject. I eventually got there, but laying out the call tree a bit differently would definitely have helped me understand what was going on much more quickly. What I realized I was looking for was a single "entry point" into caching, e.g.:

func (s *Source) cacheProject(repoUrl string, project *gitlab.Project)

When a gitlab.Project is available, this could be used directly. When only a repo URL is available, it could be transformed into a gitlab.Project using a second helper function:

func (s *Source) getGitlabProject(repoURL string) *gitlab.Project

This way, every function call that caches something looks the "same," and the function that acquires new (necessary) information for caching has a clear name. (ensure is a suboptimal function verb, because it's vague.)

What do you think of this?

ensureProjectInCache is a useful function because it does multiple things:

It first checks if the project is in cache

If not, it fetches the project from the API

It then stores the project in cache

This functionality is required in a couple of places which is why I grouped it into a single function. Breaking it down would simply mean making two separate calls in the same order at two separate places.

What I agree with is that we create the two methods cacheProject and getGitlabProject, and use it inside ensureProjectInCache. So when we're enumerating projects, we directly call cacheProject, but in the case of configured repos and ChunkUnit, we use ensureProjectInCache. What do you think?

I've gone ahead and made these changes. Let me know if you think otherwise and I'll revert.

i misread the pr when i requested changes

rosecodym

I'm bummed out that the git processing code requires this sort of contortion, but it looks like you've done it pretty cleanly. Nice work!

mustansir14 added 3 commits December 9, 2025 17:00

add wrapper reporter to append project details to chunk metadata

4b7f27a

use cache to store project details

57d1872

revert unnecessary change

bd60e85

mustansir14 requested a review from a team December 11, 2025 14:32

mustansir14 requested a review from a team as a code owner December 11, 2025 14:32

Merge branch 'main' into INS-206-Store-GitLab-project-ID-in-secret-lo…

d580fca

…cation-metadata

amanfcp approved these changes Dec 15, 2025

View reviewed changes

mustansir14 added 2 commits December 15, 2025 15:42

delete from cache when done with scanning in ChunkUnit, implement PR …

59a4944

…suggestions

Merge branch 'INS-206-Store-GitLab-project-ID-in-secret-location-meta…

1a8e764

…data' of mustansir:mustansir14/trufflehog into INS-206-Store-GitLab-project-ID-in-secret-location-metadata

mariduv approved these changes Dec 15, 2025

View reviewed changes

rosecodym requested a review from a team December 15, 2025 16:51

rosecodym previously requested changes Dec 15, 2025

View reviewed changes

mustansir14 added 2 commits December 16, 2025 15:27

query project details using repo instead of having it in source unit

422022a

revert removal of build tag

0154b4d

rosecodym reviewed Dec 16, 2025

View reviewed changes

Incorporated PR comments

5e7e238

mustansir14 requested a review from rosecodym December 18, 2025 07:44

rosecodym approved these changes Dec 19, 2025

View reviewed changes

[INS-206] Store Gitlab Project ID in secret location metadata #4601

Are you sure you want to change the base?

[INS-206] Store Gitlab Project ID in secret location metadata #4601

Uh oh!

Conversation

mustansir14 commented Dec 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description:

Checklist:

Uh oh!

amanfcp left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rosecodym left a comment

Choose a reason for hiding this comment

Uh oh!

rosecodym left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rosecodym Dec 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rosecodym left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

mustansir14 commented Dec 11, 2025 •

edited

Loading

rosecodym Dec 16, 2025 •

edited

Loading