Feat/bulk import agent

[venkataprasathbaskaran]

Hi @abhinavxd

I saw the feature request of bulk csv import of agent. So wanted to give it a try and contribute to FOSS. I have implement pretty basic stuff of import via API.

Everything happens in API call itself for now. ideally this will scale when used for large files. Where we might need to implement a background worker for process file and write report of import and have resislency in place to tackle worker craches. For start i though we can implement this basic import where they can import 100 agent in a single call.

I am raising this PR with screenshot of the API and csv used. You can give feedback and suggestions so i can improve on this.

Thanks!

[abhinavxd]

Thank you for the PR! I will check it over the weekend.

[venkataprasathbaskaran]

import_agent.mov

Sure @abhinavxd Take your time. Attaching video of sample working. Still there are areas to be improved in this PR which we can go and do once we align on basic working.

[abhinavxd]

Thanks for the PR!

I propose creating an importer package that’s initialized once on App struct, allowing uploads of more than 100 records without hitting the HTTP request timeout.

It accepts imports by namespace so status can be polled later.

Something like this-

package importer


type JobStatus struct {
    Running   bool   
    Logs      []string  
    Total     int       
    Success   int    
    Errors    int      
    StartedAt time.Time
    EndedAt   time.Time
}

type Importer struct {
    jobs map[string]*JobStatus
}

func NewImporter() *Importer {
    i := &Importer{
        jobs: make(map[string]*JobStatus),
    }
    // could start a new goroutine here that cleans up jobs every hour
    go i.cleanUp()
    return i
}

func (i *Importer) Submit(namespace string, fn func() error) error {
    // ...
    fn()
    // ...
}

On the front end, replace the import button with a common Importer.vue component that opens a dialog, lets the user select a file, use clicks import. We can poll:

GET - /api/v1/agents/import/status 

for logs and progress. This lets us reuse the importer across the app.

Let me know what you think!