Writing to a file in Golang across concurrent go routines
up vote
0
down vote
favorite
I've been reading around how golang writes to a file, and this stack overflow question and this reddit question highlights the fact Go doesn't gurantee atomicity when writing to a file system. Although I didn't see any interleaved writes (which I guess could be due to writev(2)), I didn't want to risk it so I built a a simple Go interface to do that.
I'm not super proficient with Go, but I'd like to understand how this code can be improved with best practices and on potential issues that may arise when using it.
package main
import (
"fmt"
"io"
"os"
)
// FileLogger defines the methods to log to file
type FileLogger interface {
Write(s string)
}
type fileLogger struct {
stream chan string
writer io.Writer
}
func (l *fileLogger) run() {
for {
select {
case s := <-l.stream:
_, err := l.writer.Write(byte(s))
if err != nil {
fmt.Println("Error writing to file: ", err.Error())
}
}
}
}
func (l *fileLogger) Write(s string) {
l.stream <- s
}
// NewFileLogger returns a new FileLogger
func NewFileLogger() FileLogger {
file, err := os.OpenFile(
"logs/logs.txt", os.O_RDWR|os.O_CREATE|os.O_APPEND, 0666,
)
if err != nil {
panic(err)
}
f := &fileLogger{make(chan string), file}
go f.run()
return f
}
I could for instance make it conform to the io.Writer interface, but I'm not sure what benefits there are of that?
Would there be an advantage to mutex locks here using the sync package?
file-system go interface
asked 4 hours ago
Rambatino
1442
add a comment |
up vote
0
down vote
favorite
I've been reading around how golang writes to a file, and this stack overflow question and this reddit question highlights the fact Go doesn't gurantee atomicity when writing to a file system. Although I didn't see any interleaved writes (which I guess could be due to writev(2)), I didn't want to risk it so I built a a simple Go interface to do that.
I'm not super proficient with Go, but I'd like to understand how this code can be improved with best practices and on potential issues that may arise when using it.
package main
import (
"fmt"
"io"
"os"
)
// FileLogger defines the methods to log to file
type FileLogger interface {
Write(s string)
}
type fileLogger struct {
stream chan string
writer io.Writer
}
func (l *fileLogger) run() {
for {
select {
case s := <-l.stream:
_, err := l.writer.Write(byte(s))
if err != nil {
fmt.Println("Error writing to file: ", err.Error())
}
}
}
}
func (l *fileLogger) Write(s string) {
l.stream <- s
}
// NewFileLogger returns a new FileLogger
func NewFileLogger() FileLogger {
file, err := os.OpenFile(
"logs/logs.txt", os.O_RDWR|os.O_CREATE|os.O_APPEND, 0666,
)
if err != nil {
panic(err)
}
f := &fileLogger{make(chan string), file}
go f.run()
return f
}
I could for instance make it conform to the io.Writer interface, but I'm not sure what benefits there are of that?
Would there be an advantage to mutex locks here using the sync package?
file-system go interface
asked 4 hours ago
Rambatino
1442
add a comment |
up vote
0
down vote
favorite
up vote
0
down vote
favorite
I've been reading around how golang writes to a file, and this stack overflow question and this reddit question highlights the fact Go doesn't gurantee atomicity when writing to a file system. Although I didn't see any interleaved writes (which I guess could be due to writev(2)), I didn't want to risk it so I built a a simple Go interface to do that.
I'm not super proficient with Go, but I'd like to understand how this code can be improved with best practices and on potential issues that may arise when using it.
package main
import (
"fmt"
"io"
"os"
)
// FileLogger defines the methods to log to file
type FileLogger interface {
Write(s string)
}
type fileLogger struct {
stream chan string
writer io.Writer
}
func (l *fileLogger) run() {
for {
select {
case s := <-l.stream:
_, err := l.writer.Write(byte(s))
if err != nil {
fmt.Println("Error writing to file: ", err.Error())
}
}
}
}
func (l *fileLogger) Write(s string) {
l.stream <- s
}
// NewFileLogger returns a new FileLogger
func NewFileLogger() FileLogger {
file, err := os.OpenFile(
"logs/logs.txt", os.O_RDWR|os.O_CREATE|os.O_APPEND, 0666,
)
if err != nil {
panic(err)
}
f := &fileLogger{make(chan string), file}
go f.run()
return f
}
I could for instance make it conform to the io.Writer interface, but I'm not sure what benefits there are of that?
Would there be an advantage to mutex locks here using the sync package?
file-system go interface
asked 4 hours ago
Rambatino
1442
I've been reading around how golang writes to a file, and this stack overflow question and this reddit question highlights the fact Go doesn't gurantee atomicity when writing to a file system. Although I didn't see any interleaved writes (which I guess could be due to writev(2)), I didn't want to risk it so I built a a simple Go interface to do that.
I'm not super proficient with Go, but I'd like to understand how this code can be improved with best practices and on potential issues that may arise when using it.
package main
import (
"fmt"
"io"
"os"
)
// FileLogger defines the methods to log to file
type FileLogger interface {
Write(s string)
}
type fileLogger struct {
stream chan string
writer io.Writer
}
func (l *fileLogger) run() {
for {
select {
case s := <-l.stream:
_, err := l.writer.Write(byte(s))
if err != nil {
fmt.Println("Error writing to file: ", err.Error())
}
}
}
}
func (l *fileLogger) Write(s string) {
l.stream <- s
}
// NewFileLogger returns a new FileLogger
func NewFileLogger() FileLogger {
file, err := os.OpenFile(
"logs/logs.txt", os.O_RDWR|os.O_CREATE|os.O_APPEND, 0666,
)
if err != nil {
panic(err)
}
f := &fileLogger{make(chan string), file}
go f.run()
return f
}
I could for instance make it conform to the io.Writer interface, but I'm not sure what benefits there are of that?
Would there be an advantage to mutex locks here using the sync package?
file-system go interface
file-system go interface
asked 4 hours ago
Rambatino
1442
asked 4 hours ago
Rambatino
1442
asked 4 hours ago
Rambatino
1442
asked 4 hours ago
Rambatino
1442
asked 4 hours ago
Rambatino
1442
1442
add a comment |
add a comment |
1 Answer
1
active
oldest
votes
up vote
0
down vote
Go doesn't guarantee atomicity ... so I built a simple Go interface to do that
Okay, but is your implementation atomic? The only atomic operations guaranteed to be atomic in Go are through the sync/atomic package (or things like func (*Cond) Wait under sync).
If you need true atomic write, use the atomic package. However, using log is usually sufficient.
Your implementation looks like it's trying to be concurrent, not atomic. Rather than rolling your own concurrent writer interface, use the standard API.
You're right, fmt (& os write functions, etc.) do not provide concurrency. However, the log package does.
You can see that they use mutex locks for the Output function, which is used by almost everything else.
This should perform nearly identical to your use case, because you open the file with O_APPEND, and log appends.
So open a file, and pass it to log.New().
For example:
package main
import (
"log"
"os"
)
func main() {
f, err := os.OpenFile("testfile", os.O_APPEND|os.O_CREATE|os.O_WRONLY, 0644)
if err != nil {
log.Fatal(err)
}
logger := log.New(f, "", 0)
logger.Output(2, "wow")
}
answered 4 hours ago
esote
1,7291933
thanks for the response! Is it not atomic if I have a single go routine that handles the writes using the for select?
– Rambatino
3 hours ago
I see, it's concurrent...
– Rambatino
3 hours ago
I think I have some more reading to do...
– Rambatino
3 hours ago
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
return StackExchange.using("mathjaxEditing", function () {
StackExchange.MarkdownEditor.creationCallbacks.add(function (editor, postfix) {
StackExchange.mathjaxEditing.prepareWmdForMathJax(editor, postfix, [["\$", "\$"]]);
});
});
}, "mathjax-editing");
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "196"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fcodereview.stackexchange.com%2fquestions%2f209870%2fwriting-to-a-file-in-golang-across-concurrent-go-routines%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
up vote
0
down vote
Go doesn't guarantee atomicity ... so I built a simple Go interface to do that
Okay, but is your implementation atomic? The only atomic operations guaranteed to be atomic in Go are through the sync/atomic package (or things like func (*Cond) Wait under sync).
If you need true atomic write, use the atomic package. However, using log is usually sufficient.
Your implementation looks like it's trying to be concurrent, not atomic. Rather than rolling your own concurrent writer interface, use the standard API.
You're right, fmt (& os write functions, etc.) do not provide concurrency. However, the log package does.
You can see that they use mutex locks for the Output function, which is used by almost everything else.
This should perform nearly identical to your use case, because you open the file with O_APPEND, and log appends.
So open a file, and pass it to log.New().
For example:
package main
import (
"log"
"os"
)
func main() {
f, err := os.OpenFile("testfile", os.O_APPEND|os.O_CREATE|os.O_WRONLY, 0644)
if err != nil {
log.Fatal(err)
}
logger := log.New(f, "", 0)
logger.Output(2, "wow")
}
answered 4 hours ago
esote
1,7291933
thanks for the response! Is it not atomic if I have a single go routine that handles the writes using the for select?
– Rambatino
3 hours ago
I see, it's concurrent...
– Rambatino
3 hours ago
I think I have some more reading to do...
– Rambatino
3 hours ago
add a comment |
up vote
0
down vote
Go doesn't guarantee atomicity ... so I built a simple Go interface to do that
Okay, but is your implementation atomic? The only atomic operations guaranteed to be atomic in Go are through the sync/atomic package (or things like func (*Cond) Wait under sync).
If you need true atomic write, use the atomic package. However, using log is usually sufficient.
Your implementation looks like it's trying to be concurrent, not atomic. Rather than rolling your own concurrent writer interface, use the standard API.
You're right, fmt (& os write functions, etc.) do not provide concurrency. However, the log package does.
You can see that they use mutex locks for the Output function, which is used by almost everything else.
This should perform nearly identical to your use case, because you open the file with O_APPEND, and log appends.
So open a file, and pass it to log.New().
For example:
package main
import (
"log"
"os"
)
func main() {
f, err := os.OpenFile("testfile", os.O_APPEND|os.O_CREATE|os.O_WRONLY, 0644)
if err != nil {
log.Fatal(err)
}
logger := log.New(f, "", 0)
logger.Output(2, "wow")
}
answered 4 hours ago
esote
1,7291933
thanks for the response! Is it not atomic if I have a single go routine that handles the writes using the for select?
– Rambatino
3 hours ago
I see, it's concurrent...
– Rambatino
3 hours ago
I think I have some more reading to do...
– Rambatino
3 hours ago
add a comment |
up vote
0
down vote
up vote
0
down vote
Go doesn't guarantee atomicity ... so I built a simple Go interface to do that
Okay, but is your implementation atomic? The only atomic operations guaranteed to be atomic in Go are through the sync/atomic package (or things like func (*Cond) Wait under sync).
If you need true atomic write, use the atomic package. However, using log is usually sufficient.
Your implementation looks like it's trying to be concurrent, not atomic. Rather than rolling your own concurrent writer interface, use the standard API.
You're right, fmt (& os write functions, etc.) do not provide concurrency. However, the log package does.
You can see that they use mutex locks for the Output function, which is used by almost everything else.
This should perform nearly identical to your use case, because you open the file with O_APPEND, and log appends.
So open a file, and pass it to log.New().
For example:
package main
import (
"log"
"os"
)
func main() {
f, err := os.OpenFile("testfile", os.O_APPEND|os.O_CREATE|os.O_WRONLY, 0644)
if err != nil {
log.Fatal(err)
}
logger := log.New(f, "", 0)
logger.Output(2, "wow")
}
answered 4 hours ago
esote
1,7291933
Go doesn't guarantee atomicity ... so I built a simple Go interface to do that
Okay, but is your implementation atomic? The only atomic operations guaranteed to be atomic in Go are through the sync/atomic package (or things like func (*Cond) Wait under sync).
If you need true atomic write, use the atomic package. However, using log is usually sufficient.
Your implementation looks like it's trying to be concurrent, not atomic. Rather than rolling your own concurrent writer interface, use the standard API.
You're right, fmt (& os write functions, etc.) do not provide concurrency. However, the log package does.
You can see that they use mutex locks for the Output function, which is used by almost everything else.
This should perform nearly identical to your use case, because you open the file with O_APPEND, and log appends.
So open a file, and pass it to log.New().
For example:
package main
import (
"log"
"os"
)
func main() {
f, err := os.OpenFile("testfile", os.O_APPEND|os.O_CREATE|os.O_WRONLY, 0644)
if err != nil {
log.Fatal(err)
}
logger := log.New(f, "", 0)
logger.Output(2, "wow")
}
answered 4 hours ago
esote
1,7291933
edited 3 hours ago
answered 4 hours ago
esote
1,7291933
answered 4 hours ago
esote
1,7291933
answered 4 hours ago
esote
1,7291933
1,7291933
thanks for the response! Is it not atomic if I have a single go routine that handles the writes using the for select?
– Rambatino
3 hours ago
I see, it's concurrent...
– Rambatino
3 hours ago
I think I have some more reading to do...
– Rambatino
3 hours ago
add a comment |
thanks for the response! Is it not atomic if I have a single go routine that handles the writes using the for select?
– Rambatino
3 hours ago
I see, it's concurrent...
– Rambatino
3 hours ago
I think I have some more reading to do...
– Rambatino
3 hours ago
thanks for the response! Is it not atomic if I have a single go routine that handles the writes using the for select?
– Rambatino
3 hours ago
thanks for the response! Is it not atomic if I have a single go routine that handles the writes using the for select?
– Rambatino
3 hours ago
I see, it's concurrent...
– Rambatino
3 hours ago
I see, it's concurrent...
– Rambatino
3 hours ago
I think I have some more reading to do...
– Rambatino
3 hours ago
I think I have some more reading to do...
– Rambatino
3 hours ago
add a comment |
Thanks for contributing an answer to Code Review Stack Exchange!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
Use MathJax to format equations. MathJax reference.
To learn more, see our tips on writing great answers.
Some of your past answers have not been well-received, and you're in danger of being blocked from answering.
Please pay close attention to the following guidance:
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fcodereview.stackexchange.com%2fquestions%2f209870%2fwriting-to-a-file-in-golang-across-concurrent-go-routines%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
