CODEWITHSUNDEEP
kedrogithub-actions-self-hosted-runners
magical_unicorn
magical_unicorn

Reputation: 11

how do I run a local script using github actions

Hello I am using kedro (a pipeline tool) and want to use github actions to trigger a kedro command (kedro run) whenever I make a push to my github repo.

Since I have all the data in my local repo, I thought it would make sense to run the kedro command on my local machine. So my question is, is there a way to trigger a local action using github actions? Using self-hosted runners perhaps?

Upvotes: 1

Views: 580

Answers (2)

datajoely
datajoely

Reputation: 1516

Hi @magical_unicorn I think @avan-sh's answer is correct - what I would also add is that we encourage you to not commit data to VCS / Git. There are some technical limitations such as filesize, but more importantly it's not great security practice when working with teams.

Whilst not necessary on all projects - it might be good practice to explore using some sort of cloud storage for your data, decoupling it from the version control system.

Upvotes: 0

avan-sh
avan-sh

Reputation: 11

You can run the kedro pipeline directly in gh provided runner using the steps below. I added a script I used previously in here to run kedro lint with every push.

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v2
    - name: Set up Python 3.7.9
      uses: actions/setup-python@v2
      with:
        python-version: 3.7.9
    - uses: actions/cache@v2
      with:
        path: ${{ env.pythonLocation }}
        key: ${{ env.pythonLocation }}-${{ hashFiles('src/requirements.txt') }}

    - name: Install dependencies
      run: |
        python -m pip install --upgrade pip
        pip install -r src/requirements.txt
    - name: Run Kedro Pipeline
      run: |
        kedro run

That said, I'm also wondering why you would need to run it on every push. Given compute provided by github actions is likely resource constrained running there might not be the best place. You would also need to keep your data within your repo for this approach to work.

Upvotes: 1

Related Questions