15 Commits
0.7.1 ... main

Author SHA1 Message Date
Anton
4411aef6fc doc: api - fix broken link 2024-06-13 03:04:31 +05:00
Anton
2a95a86a2f doc: api 2024-06-13 03:03:31 +05:00
Anton
e0b733b18c doc: readme - funding 2024-06-13 03:01:09 +05:00
Anton Zhiyanov
d7e55541c3 doc: funding 2024-06-08 23:43:48 +05:00
Andreas Deininger
f250034cd4 infr: bump github action workflows (#11) 2024-03-28 09:12:01 +05:00
Andreas Deininger
38dfe1a380 fix: typos in comments (#10) 2024-03-28 09:10:03 +05:00
Anton
e6c7b053f9 doc: codapi 0.8.0 2024-03-11 00:12:07 +05:00
Anton
4468c6193c refactor: trim space in stdout and stderr 2024-03-03 20:13:26 +05:00
Anton
49dffc8f1d feat: docker stop action 2024-03-03 16:53:29 +05:00
Anton
d6945f0048 impr: backward-compatible compliance with rfc 2397 2024-02-20 12:10:21 +05:00
Anton
460493eeaa impr: comply with rfc 2397 for binary files in request 2024-02-20 11:46:06 +05:00
Stanislav Fesenko
cc3567f26e impr: unlimited replacements when expanding command vars (#8) 2024-02-20 11:30:22 +05:00
Anton
4218065e0e build: build on pull request 2024-02-20 11:22:19 +05:00
Anton
12f7e25a85 impr: request files have priority over box files 2024-02-11 18:23:37 +05:00
Anton
b312100bbc doc: codapi 0.7.1 2024-01-19 21:01:51 +05:00
16 changed files with 364 additions and 129 deletions

1
.github/FUNDING.yml vendored Normal file
View File

@@ -0,0 +1 @@
github: [nalgeon]

View File

@@ -7,6 +7,12 @@ on:
- "docs/**"
- Makefile
- README.md
pull_request:
branches: [main]
paths-ignore:
- "docs/**"
- Makefile
- README.md
workflow_dispatch:
defaults:
@@ -18,10 +24,10 @@ jobs:
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v3
uses: actions/checkout@v4
- name: Setup Go
uses: actions/setup-go@v4
uses: actions/setup-go@v5
with:
go-version: "stable"
@@ -29,7 +35,7 @@ jobs:
run: make test build
- name: Upload artifact
uses: actions/upload-artifact@v3
uses: actions/upload-artifact@v4
with:
name: codapi
path: build/codapi

View File

@@ -14,15 +14,15 @@ jobs:
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v3
uses: actions/checkout@v4
- name: Setup Go
uses: actions/setup-go@v4
uses: actions/setup-go@v5
with:
go-version: "stable"
- name: Release and publish
uses: goreleaser/goreleaser-action@v4
uses: goreleaser/goreleaser-action@v5
with:
args: release --clean
env:

View File

@@ -32,51 +32,11 @@ For an introduction to Codapi, see this post: [Interactive code examples for fun
See [Installing Codapi](docs/install.md) for details.
## Usage (API)
## Usage
Call `/v1/exec` to run the code in a sandbox:
See [API](docs/api.md) to run sandboxed code using the HTTP API.
```http
POST https://api.codapi.org/v1/exec
content-type: application/json
{
"sandbox": "python",
"command": "run",
"files": {
"": "print('hello world')"
}
}
```
`sandbox` is the name of the pre-configured sandbox, and `command` is the name of a command supported by that sandbox. See [Adding a sandbox](docs/add-sandbox.md) for details on how to add a new sandbox.
`files` is a map, where the key is a filename and the value is its contents. When executing a single file, it should either be named as the `command` expects, or be an empty string (as in the example above).
Response:
```http
HTTP/1.1 200 OK
Content-Type: application/json
```
- `id` is the unique execution identifier.
- `ok` is `true` if the code executed without errors, or `false` otherwise.
- `duration` is the execution time in milliseconds.
- `stdout` is what the code printed to the standard output.
- `stderr` is what the code printed to the standard error, or a compiler/os error (if any).
## Usage (JavaScript)
See [codapi-js](https://github.com/nalgeon/codapi-js) to embed the JavaScript widget into a web page. The widget uses exactly the same API as described above.
## Contributing
Contributions are welcome. For anything other than bugfixes, please first open an issue to discuss what you want to change.
Be sure to add or update tests as appropriate.
See [codapi-js](https://github.com/nalgeon/codapi-js) to embed the JavaScript widget into a web page.
## Contributing
@@ -84,12 +44,12 @@ Contributions are welcome. For anything other than bugfixes, please first open a
Be sure to add or update tests as appropriate.
## Funding
Copyright 2023 [Anton Zhiyanov](https://antonz.org/).
Codapi is mostly a [one-man](https://antonz.org/) project, not backed by a VC fund or anything.
The software is available under the Apache-2.0 license.
If you find Codapi useful, please consider sponsoring it on GitHub. It really helps to move the project forward.
## Stay tuned
♥ [Become a sponsor](https://github.com/sponsors/nalgeon) to support Codapi.
★ [Subscribe](https://antonz.org/subscribe/) to stay on top of new features.

42
docs/api.md Normal file
View File

@@ -0,0 +1,42 @@
# API
Call `/v1/exec` to run the code in a sandbox:
```http
POST https://api.codapi.org/v1/exec
content-type: application/json
{
"sandbox": "python",
"command": "run",
"files": {
"": "print('hello world')"
}
}
```
`sandbox` is the name of the pre-configured sandbox, and `command` is the name of a command supported by that sandbox. See [Adding a sandbox](add-sandbox.md) for details on how to add a new sandbox.
`files` is a map, where the key is a filename and the value is its contents. When executing a single file, it should either be named as the `command` expects, or be an empty string (as in the example above).
Response:
```http
HTTP/1.1 200 OK
Content-Type: application/json
```
- `id` is the unique execution identifier.
- `ok` is `true` if the code executed without errors, or `false` otherwise.
- `duration` is the execution time in milliseconds.
- `stdout` is what the code printed to the standard output.
- `stderr` is what the code printed to the standard error, or a compiler/os error (if any).
-
- `id` is the unique execution identifier.
- `ok` is `true` if the code executed without errors, or `false` otherwise.
- `duration` is the execution time in milliseconds.
- `stdout` is what the code printed to the standard output.
- `stderr` is what the code printed to the standard error, or a compiler/os error (if any).
-

View File

@@ -28,10 +28,10 @@ docker run hello-world
```sh
cd /opt/codapi
curl -L -O "https://github.com/nalgeon/codapi/releases/download/0.6.0/codapi_0.6.0_linux_amd64.tar.gz"
tar xvzf codapi_0.6.0_linux_amd64.tar.gz
curl -L -O "https://github.com/nalgeon/codapi/releases/download/0.8.0/codapi_0.8.0_linux_amd64.tar.gz"
tar xvzf codapi_0.8.0_linux_amd64.tar.gz
chmod +x codapi
rm -f codapi_0.6.0_linux_amd64.tar.gz
rm -f codapi_0.8.0_linux_amd64.tar.gz
```
5. Build Docker images (as codapi):

View File

@@ -6,7 +6,7 @@ import (
"sort"
)
// A Config describes application cofig.
// A Config describes application config.
type Config struct {
PoolSize int `json:"pool_size"`
Verbose bool `json:"verbose"`
@@ -99,6 +99,7 @@ type Step struct {
Version string `json:"version"`
User string `json:"user"`
Action string `json:"action"`
Detach bool `json:"detach"`
Stdin bool `json:"stdin"`
Command []string `json:"command"`
Timeout int `json:"timeout"`

View File

@@ -23,6 +23,7 @@ var killTimeout = 5 * time.Second
const (
actionRun = "run"
actionExec = "exec"
actionStop = "stop"
)
// A Docker engine executes a specific sandbox command
@@ -125,9 +126,9 @@ func (e *Docker) execStep(step *config.Step, req Request, dir string, files File
// getBox selects an appropriate box for the step (if any).
func (e *Docker) getBox(step *config.Step, req Request) (*config.Box, error) {
if step.Action == actionExec {
// exec steps use existing instances
// and do not spin up new boxes
if step.Action != actionRun {
// steps other than "run" use existing containers
// and do not spin up new ones
return nil, nil
}
var boxName string
@@ -208,7 +209,7 @@ func (e *Docker) exec(box *config.Box, step *config.Step, req Request, dir strin
if err.Error() == "signal: killed" {
if step.Action == actionRun {
// we have to "docker kill" the container here, because the proccess
// we have to "docker kill" the container here, because the process
// inside the container is not related to the "docker run" process,
// and will hang forever after the "docker run" process is killed
go func() {
@@ -244,11 +245,14 @@ func (e *Docker) exec(box *config.Box, step *config.Step, req Request, dir strin
// buildArgs prepares the arguments for the `docker` command.
func (e *Docker) buildArgs(box *config.Box, step *config.Step, req Request, dir string) []string {
var args []string
if step.Action == actionRun {
switch step.Action {
case actionRun:
args = dockerRunArgs(box, step, req, dir)
} else if step.Action == actionExec {
args = dockerExecArgs(step)
} else {
case actionExec:
args = dockerExecArgs(step, req)
case actionStop:
args = dockerStopArgs(step, req)
default:
// should never happen if the config is valid
args = []string{"version"}
}
@@ -271,12 +275,15 @@ func dockerRunArgs(box *config.Box, step *config.Step, req Request, dir string)
"--pids-limit", strconv.Itoa(box.NProc),
"--user", step.User,
}
if !box.Writable {
args = append(args, "--read-only")
if step.Detach {
args = append(args, "--detach")
}
if step.Stdin {
args = append(args, "--interactive")
}
if !box.Writable {
args = append(args, "--read-only")
}
if box.Storage != "" {
args = append(args, "--storage-opt", fmt.Sprintf("size=%s", box.Storage))
}
@@ -300,14 +307,23 @@ func dockerRunArgs(box *config.Box, step *config.Step, req Request, dir string)
}
// dockerExecArgs prepares the arguments for the `docker exec` command.
func dockerExecArgs(step *config.Step) []string {
func dockerExecArgs(step *config.Step, req Request) []string {
// :name means executing in the container passed in the request
box := strings.Replace(step.Box, ":name", req.ID, 1)
return []string{
actionExec, "--interactive",
"--user", step.User,
step.Box,
box,
}
}
// dockerStopArgs prepares the arguments for the `docker stop` command.
func dockerStopArgs(step *config.Step, req Request) []string {
// :name means executing in the container passed in the request
box := strings.Replace(step.Box, ":name", req.ID, 1)
return []string{actionStop, box}
}
// filesReader creates a reader over an in-memory collection of files.
func filesReader(files Files) io.Reader {
var input strings.Builder
@@ -323,7 +339,7 @@ func expandVars(command []string, name string) []string {
expanded := make([]string, len(command))
copy(expanded, command)
for i, cmd := range expanded {
expanded[i] = strings.Replace(cmd, ":name", name, 1)
expanded[i] = strings.Replace(cmd, ":name", name, -1)
}
return expanded
}

View File

@@ -59,6 +59,27 @@ var dockerCfg = &config.Config{
},
},
Commands: map[string]config.SandboxCommands{
"alpine": map[string]*config.Command{
"echo": {
Engine: "docker",
Before: &config.Step{
Box: "alpine", User: "sandbox", Action: "run", Detach: true,
Command: []string{"echo", "before"},
NOutput: 4096,
},
Steps: []*config.Step{
{
Box: ":name", User: "sandbox", Action: "exec",
Command: []string{"sh", "main.sh"},
NOutput: 4096,
},
},
After: &config.Step{
Box: ":name", User: "sandbox", Action: "stop",
NOutput: 4096,
},
},
},
"go": map[string]*config.Command{
"run": {
Engine: "docker",
@@ -297,11 +318,55 @@ func TestDockerExec(t *testing.T) {
})
}
func TestDockerStop(t *testing.T) {
logx.Mock()
commands := map[string]execy.CmdOut{
"docker run": {Stdout: "c958ff2", Stderr: "", Err: nil},
"docker exec": {Stdout: "hello", Stderr: "", Err: nil},
"docker stop": {Stdout: "alpine_42", Stderr: "", Err: nil},
}
mem := execy.Mock(commands)
engine := NewDocker(dockerCfg, "alpine", "echo")
t.Run("success", func(t *testing.T) {
req := Request{
ID: "alpine_42",
Sandbox: "alpine",
Command: "echo",
Files: map[string]string{
"": "echo hello",
},
}
out := engine.Exec(req)
if out.ID != req.ID {
t.Errorf("ID: expected %s, got %s", req.ID, out.ID)
}
if !out.OK {
t.Error("OK: expected true")
}
want := "hello"
if out.Stdout != want {
t.Errorf("Stdout: expected %q, got %q", want, out.Stdout)
}
if out.Stderr != "" {
t.Errorf("Stderr: expected %q, got %q", "", out.Stdout)
}
if out.Err != nil {
t.Errorf("Err: expected nil, got %v", out.Err)
}
mem.MustHave(t, "docker run --rm --name alpine_42", "--detach")
mem.MustHave(t, "docker exec --interactive --user sandbox alpine_42 sh main.sh")
mem.MustHave(t, "docker stop alpine_42")
})
}
func Test_expandVars(t *testing.T) {
const name = "codapi_01"
commands := map[string]string{
"python main.py": "python main.py",
"sh create.sh :name": "sh create.sh " + name,
"sh copy.sh :name new-:name": "sh copy.sh " + name + " new-" + name,
}
for cmd, want := range commands {
src := strings.Fields(cmd)

View File

@@ -63,7 +63,7 @@ func (err ExecutionError) Unwrap() error {
}
// An ArgumentError is returned if code execution failed
// due to the invalid value of the request agrument.
// due to the invalid value of the request argument.
type ArgumentError struct {
name string
reason error

View File

@@ -48,7 +48,7 @@ func (p *Program) RunStdin(stdin io.Reader, id, name string, arg ...string) (std
cmd.Stdout = LimitWriter(&cmdout, p.nOutput)
cmd.Stderr = LimitWriter(&cmderr, p.nOutput)
err = execy.Run(cmd)
stdout = cmdout.String()
stderr = cmderr.String()
stdout = strings.TrimSpace(cmdout.String())
stderr = strings.TrimSpace(cmderr.String())
return
}

View File

@@ -12,8 +12,17 @@ import (
"strings"
)
// Exists checks if the specified path exists.
func Exists(path string) bool {
_, err := os.Stat(path)
// we need a double negation here, because
// errors.Is(err, os.ErrExist)
// does not work
return !errors.Is(err, os.ErrNotExist)
}
// CopyFile copies all files matching the pattern
// to the destination directory.
// to the destination directory. Does not overwrite existing file.
func CopyFiles(pattern string, dstDir string, perm fs.FileMode) error {
matches, err := filepath.Glob(pattern)
if err != nil {
@@ -28,6 +37,10 @@ func CopyFiles(pattern string, dstDir string, perm fs.FileMode) error {
defer src.Close()
dstFile := filepath.Join(dstDir, filepath.Base(match))
if Exists(dstFile) {
continue
}
dst, err := os.OpenFile(dstFile, os.O_RDWR|os.O_CREATE|os.O_TRUNC, perm)
if err != nil {
return err
@@ -62,20 +75,29 @@ func ReadJson[T any](path string) (T, error) {
// e.g. data:application/octet-stream;base64,MTIz
func WriteFile(path, content string, perm fs.FileMode) (err error) {
var data []byte
if strings.HasPrefix(content, "data:") {
if !strings.HasPrefix(content, "data:") {
// text file
data = []byte(content)
return os.WriteFile(path, data, perm)
}
// data-url encoded file
_, encoded, found := strings.Cut(content, ",")
meta, encoded, found := strings.Cut(content, ",")
if !found {
return errors.New("invalid data-url encoding")
}
if !strings.HasSuffix(meta, "base64") {
// no need to decode
data = []byte(encoded)
return os.WriteFile(path, data, perm)
}
// decode base64-encoded data
data, err = base64.StdEncoding.DecodeString(encoded)
if err != nil {
return err
}
} else {
// text file
data = []byte(content)
}
return os.WriteFile(path, data, perm)
}

View File

@@ -8,29 +8,49 @@ import (
"testing"
)
func TestExists(t *testing.T) {
t.Run("exists", func(t *testing.T) {
path := filepath.Join(t.TempDir(), "file.txt")
err := os.WriteFile(path, []byte{1, 2, 3}, 0444)
if err != nil {
t.Fatal(err)
}
if !Exists(path) {
t.Fatalf("Exists: %s does not exist", filepath.Base(path))
}
})
t.Run("does not exist", func(t *testing.T) {
path := filepath.Join(t.TempDir(), "file.txt")
if Exists(path) {
t.Fatalf("Exists: %s should not exist", filepath.Base(path))
}
})
}
func TestCopyFiles(t *testing.T) {
// Create a temporary directory for testing
// create a temporary directory for testing
srcDir, err := os.MkdirTemp("", "src")
if err != nil {
t.Fatal(err)
}
defer os.RemoveAll(srcDir)
// Create a source file
// create a source file
srcFile := filepath.Join(srcDir, "source.txt")
err = os.WriteFile(srcFile, []byte("test data"), 0644)
if err != nil {
t.Fatal(err)
}
// Specify the destination directory
// specify the destination directory
dstDir, err := os.MkdirTemp("", "dst")
if err != nil {
t.Fatal(err)
}
defer os.RemoveAll(dstDir)
// Call the CopyFiles function
t.Run("copy", func(t *testing.T) {
// call the CopyFiles function
const perm = fs.FileMode(0444)
pattern := filepath.Join(srcDir, "*.txt")
err = CopyFiles(pattern, dstDir, perm)
@@ -38,7 +58,7 @@ func TestCopyFiles(t *testing.T) {
t.Fatal(err)
}
// Verify that the file was copied correctly
// verify that the file was copied correctly
dstFile := filepath.Join(dstDir, "source.txt")
fileInfo, err := os.Stat(dstFile)
if err != nil {
@@ -48,17 +68,59 @@ func TestCopyFiles(t *testing.T) {
t.Errorf("unexpected file permissions: got %v, want %v", fileInfo.Mode(), perm)
}
// Read the contents of the copied file
// read the contents of the copied file
data, err := os.ReadFile(dstFile)
if err != nil {
t.Fatal(err)
}
// Verify the contents of the copied file
// verify the contents of the copied file
expected := []byte("test data")
if string(data) != string(expected) {
t.Errorf("unexpected file content: got %q, want %q", data, expected)
}
})
t.Run("skip existing", func(t *testing.T) {
// existing file in the destination dir
path := filepath.Join(dstDir, "existing.txt")
err := os.WriteFile(path, []byte("v1"), 0444)
if err != nil {
t.Fatal(err)
}
// same file in the source dir
path = filepath.Join(srcDir, "existing.txt")
err = os.WriteFile(path, []byte("v2"), 0444)
if err != nil {
t.Fatal(err)
}
// copy files
pattern := filepath.Join(srcDir, "*.txt")
err = CopyFiles(pattern, dstDir, 0444)
if err != nil {
t.Fatal(err)
}
// verify that the new file was copied correctly
newFile := filepath.Join(dstDir, "source.txt")
_, err = os.Stat(newFile)
if err != nil {
t.Fatalf("new file not copied: %s", err)
}
// verify that the existing file remained unchanged
existFile := filepath.Join(dstDir, "existing.txt")
data, err := os.ReadFile(existFile)
if err != nil {
t.Fatal(err)
}
expected := []byte("v1")
if string(data) != string(expected) {
t.Error("existing file got overwritten")
}
})
}
func TestReadJson(t *testing.T) {
@@ -111,8 +173,8 @@ func TestWriteFile(t *testing.T) {
}
})
t.Run("binary", func(t *testing.T) {
path := filepath.Join(dir, "data.bin")
t.Run("data-octet-stream", func(t *testing.T) {
path := filepath.Join(dir, "data-1.bin")
err = WriteFile(path, "data:application/octet-stream;base64,MTIz", 0444)
if err != nil {
t.Fatalf("expected nil err, got %v", err)
@@ -127,6 +189,38 @@ func TestWriteFile(t *testing.T) {
}
})
t.Run("data-base64", func(t *testing.T) {
path := filepath.Join(dir, "data-2.bin")
err = WriteFile(path, "data:;base64,MTIz", 0444)
if err != nil {
t.Fatalf("expected nil err, got %v", err)
}
got, err := os.ReadFile(path)
if err != nil {
t.Fatalf("read file: expected nil err, got %v", err)
}
want := []byte("123")
if !reflect.DeepEqual(got, want) {
t.Errorf("read file: expected %v, got %v", want, got)
}
})
t.Run("data-text-plain", func(t *testing.T) {
path := filepath.Join(dir, "data-3.bin")
err = WriteFile(path, "data:text/plain;,123", 0444)
if err != nil {
t.Fatalf("expected nil err, got %v", err)
}
got, err := os.ReadFile(path)
if err != nil {
t.Fatalf("read file: expected nil err, got %v", err)
}
want := []byte("123")
if !reflect.DeepEqual(got, want) {
t.Errorf("read file: expected %v, got %v", want, got)
}
})
t.Run("perm", func(t *testing.T) {
const perm = 0444
path := filepath.Join(dir, "perm.txt")
@@ -153,7 +247,7 @@ func TestWriteFile(t *testing.T) {
t.Run("invalid binary value", func(t *testing.T) {
path := filepath.Join(dir, "data.bin")
err = WriteFile(path, "data:application/octet-stream;base64,12345", 0444)
err = WriteFile(path, "data:;base64,12345", 0444)
if err == nil {
t.Fatal("expected error, got nil")
}

View File

@@ -30,9 +30,16 @@ func (m *Memory) WriteString(s string) {
}
// Has returns true if the memory has the message.
func (m *Memory) Has(msg string) bool {
func (m *Memory) Has(message ...string) bool {
for _, line := range m.Lines {
if strings.Contains(line, msg) {
containsAll := true
for _, part := range message {
if !strings.Contains(line, part) {
containsAll = false
break
}
}
if containsAll {
return true
}
}
@@ -40,20 +47,22 @@ func (m *Memory) Has(msg string) bool {
}
// MustHave checks if the memory has the message.
func (m *Memory) MustHave(t *testing.T, msg string) {
if !m.Has(msg) {
t.Errorf("%s must have: %s", m.Name, msg)
// If the message consists of several parts,
// they must all be in the same memory line.
func (m *Memory) MustHave(t *testing.T, message ...string) {
if !m.Has(message...) {
t.Errorf("%s must have: %v", m.Name, message)
}
}
// MustNotHave checks if the memory does not have the message.
func (m *Memory) MustNotHave(t *testing.T, msg string) {
if m.Has(msg) {
t.Errorf("%s must NOT have: %s", m.Name, msg)
func (m *Memory) MustNotHave(t *testing.T, message ...string) {
if m.Has(message...) {
t.Errorf("%s must NOT have: %v", m.Name, message)
}
}
// Clear cleares the memory.
// Clear clears the memory.
func (m *Memory) Clear() {
m.Lines = []string{}
}

View File

@@ -40,4 +40,23 @@ func TestMemory_Has(t *testing.T) {
if !mem.Has("hello world") {
t.Error("Has: unexpected false")
}
_, _ = mem.Write([]byte("one two three four"))
if !mem.Has("one two") {
t.Error("Has: one two: unexpected false")
}
if !mem.Has("two three") {
t.Error("Has: two three: unexpected false")
}
if mem.Has("one three") {
t.Error("Has: one three: unexpected true")
}
if !mem.Has("one", "three") {
t.Error("Has: [one three]: unexpected false")
}
if !mem.Has("one", "three", "four") {
t.Error("Has: [one three four]: unexpected false")
}
if !mem.Has("four", "three") {
t.Error("Has: [four three]: unexpected false")
}
}

View File

@@ -20,7 +20,7 @@ var engineConstr = map[string]func(*config.Config, string, string) engine.Engine
}
// engines is the registry of command executors.
// Each engine executes a specific command in a specifix sandbox.
// Each engine executes a specific command in a specific sandbox.
// sandbox : command : engine
// TODO: Maybe it's better to create a single instance of each engine
// and pass the sandbox and command as arguments to the Exec.